You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "djvanderlaan (via GitHub)" <gi...@apache.org> on 2025/05/13 15:29:34 UTC
[I] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
djvanderlaan opened a new issue, #46428:
URL: https://github.com/apache/arrow/issues/46428
### Describe the bug, including details regarding any error messages, version, and platform.
I noticed that some operations use substantially slower and use more memory under arrow V20.0.0 and v19.0.0 than under v17.0.0. I managed to reduce the example and am able to reproduce this both on a production machine running ubuntu 22.04 and my home desktop (running debian stable).
The run of the example on my desktop with v17 took 10s and a maximum of approx 7GB memory. The v20 run was killed after 1m16s because it ran out of memory (my home machine is unfortunately limited to 24GB). Before being killed the memory use peaked at approx 22GB. See below for the output.
The following code generates the example data:
```r
nvert <- 10E6
nedge <- 20E7
vert <- data.frame(
id = seq_len(nvert)
)
edges <- data.frame(
src = sample(nvert, nedge, replace = TRUE),
dst = sample(nvert, nedge, replace = TRUE),
type = sample(1:10, nedge, replace = TRUE)
)
library(arrow)
write_parquet(vert, "vertices.parquet")
write_parquet(edges, "edges.parquet")
```
The following script processes this data and shows substantial differences between v20 and v17 (the production machine had v19 which showed the same behaviour are v20).
```r
library(arrow)
library(dplyr)
sessionInfo()
vert <- read_parquet("vertices.parquet")
str(vert)
con <- open_dataset("edges.parquet")
con
dta <- con |> filter(src %in% vert$id, dst %in% vert$id) |> collect()
nrow(dta)
```
Below the results for v17:
```
$ /usr/bin/time -v R --no-save < test.R
R version 4.5.0 (2025-04-11) -- "How About a Twenty-Six"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>
>
> library(arrow)
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
> library(dplyr)
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
>
> sessionInfo()
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 12 (bookworm)
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.11.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.11.0 LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=nl_NL.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=nl_NL.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=nl_NL.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C
time zone: Europe/Amsterdam
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dplyr_1.1.4 arrow_17.0.0
loaded via a namespace (and not attached):
[1] assertthat_0.2.1 R6_2.6.1 bit_4.6.0 tidyselect_1.2.1
[5] magrittr_2.0.3 glue_1.8.0 tibble_3.2.1 pkgconfig_2.0.3
[9] bit64_4.6.0-1 generics_0.1.3 lifecycle_1.0.4 cli_3.6.5
[13] vctrs_0.6.5 compiler_4.5.0 purrr_1.0.4 pillar_1.10.2
[17] rlang_1.1.6
>
> vert <- read_parquet("vertices.parquet")
> str(vert)
tibble [10,000,000 × 1] (S3: tbl_df/tbl/data.frame)
$ id: int [1:10000000] 1 2 3 4 5 6 7 8 9 10 ...
>
> con <- open_dataset("edges.parquet")
> con
FileSystemDataset with 1 Parquet file
3 columns
src: int32
dst: int32
type: int32
See $metadata for additional Schema metadata
>
> dta <- con |> filter(src %in% vert$id, dst %in% vert$id) |> collect()
>
> nrow(dta)
[1] 200000000
>
>
Command being timed: "R --no-save"
User time (seconds): 30.19
System time (seconds): 2.74
Percent of CPU this job got: 306%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.75
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 7078692
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 443
Minor (reclaiming a frame) page faults: 61785
Voluntary context switches: 10143
Involuntary context switches: 10536
Swaps: 0
File system inputs: 3898808
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
Below the results for v20:
```
$ /usr/bin/time -v R --no-save < test.R
R version 4.5.0 (2025-04-11) -- "How About a Twenty-Six"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>
>
> library(arrow)
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
> library(dplyr)
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
>
> sessionInfo()
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 12 (bookworm)
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.11.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.11.0 LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=nl_NL.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=nl_NL.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=nl_NL.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C
time zone: Europe/Amsterdam
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dplyr_1.1.4 arrow_20.0.0
loaded via a namespace (and not attached):
[1] assertthat_0.2.1 R6_2.6.1 bit_4.6.0 tidyselect_1.2.1
[5] magrittr_2.0.3 glue_1.8.0 tibble_3.2.1 pkgconfig_2.0.3
[9] bit64_4.6.0-1 generics_0.1.3 lifecycle_1.0.4 cli_3.6.5
[13] vctrs_0.6.5 compiler_4.5.0 purrr_1.0.4 pillar_1.10.2
[17] rlang_1.1.6
>
> vert <- read_parquet("vertices.parquet")
> str(vert)
tibble [10,000,000 × 1] (S3: tbl_df/tbl/data.frame)
$ id: int [1:10000000] 1 2 3 4 5 6 7 8 9 10 ...
>
> con <- open_dataset("edges.parquet")
> con
FileSystemDataset with 1 Parquet file
3 columns
src: int32
dst: int32
type: int32
See $metadata for additional Schema metadata
>
> dta <- con |> filter(src %in% vert$id, dst %in% vert$id) |> collect()
Command terminated by signal 9
Command being timed: "R --no-save"
User time (seconds): 50.57
System time (seconds): 8.59
Percent of CPU this job got: 78%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.57
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21869392
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 622
Minor (reclaiming a frame) page faults: 2185479
Voluntary context switches: 2431
Involuntary context switches: 1599
Swaps: 0
File system inputs: 140744
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
So the run with v17 took 10s and a maximum of approx 7GB memory. The v20 run was killed after 1m16s because it ran out of memory (my home machine is unfortunately limited to 24GB). Before being killed the memory use peaked at approx 22GB.
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3019350091
this is marked as a release blocker, is it?
@assignUser @amoeba @thisisnic @jonkeane
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3026798570
Findings from debugging:
* change occurred between 19.0.1.1 and 20.0.0 and results in out-of-memory error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3026537113
FYI folks I managed to replicate this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3019402776
Oh yeah I flagged this as it sounds concerning but it was in the evening so I forgot to follow up, sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3053528413
I tested with the tip of maint-21.0.0 and the issue appears to have been fixed by https://github.com/apache/arrow/issues/46777.
The script now takes ~17 sec of wall time and peaks at 7GiB of memory (on an M4 Pro) which seems about right. With arrow R 20.0.0, the script gets killed after 122s and using >120GiB of RAM on my system.
I'll remove the label and mark this as resolved. Thanks for the report @djvanderlaan, it's appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3026938038
Looks like this may be the same underlying cause as #46777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3020550793
@djvanderlaan you mentioned other operations in your original report, can you easily share those too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3026673764
Thanks @djvanderlaan, that useful extra context for us to look into!
I've put a repo together to set up quick testing, will try with a few different Arrow versions now: https://github.com/thisisnic/arrow-version-test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3053462525
Good catch @raulcd, I'll test now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "djvanderlaan (via GitHub)" <gi...@apache.org>.
djvanderlaan commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3026659445
Sorry, I wasn't clear. I have only seen this in this specific case. I wanted to indicate that I wasn't sure which operations were affected. Sorry for the confusion.
I also forgot to mention: in v18 (if I remember correctly) the `%` operator was changed. That might be related. I did look at the code changes but didn't directly spot something that might cause this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba closed issue #46428: [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations
URL: https://github.com/apache/arrow/issues/46428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] The arrow R package v20 and v19 are substantially slower and use more memory than v17 for some operations [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46428:
URL: https://github.com/apache/arrow/issues/46428#issuecomment-3051848811
This is still marked as a blocker. #46777 has been merged, can someone validate the issue is solved or should we remove the blocker label?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org