[jira] [Assigned] (ARROW-14736) [C++][R]Opening a multi-file dataset and writing a re-partitioned version of it fails

["Todd Farmer (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-14736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Farmer reassigned ARROW-14736:
-----------------------------------

    Assignee:     (was: Weston Pace)

This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

> [C++][R]Opening a multi-file dataset and writing a re-partitioned version of it fails
> -------------------------------------------------------------------------------------
>
>                 Key: ARROW-14736
>                 URL: https://issues.apache.org/jira/browse/ARROW-14736
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, R
>    Affects Versions: 6.0.0
>         Environment: M1 Mac, macOS Monterey 12.0.1, 16Gb RAM
> R 4.1.1, {arrow} R package 6.0.0.2 (release) & 6.0.0.9000 (dev)
>            Reporter: Dragoș Moldovan-Grünfeld
>            Priority: Major
>              Labels: dataset
>         Attachments: image-2021-11-17-14-43-37-127.png, image-2021-11-17-14-54-42-747.png, image-2021-11-17-14-55-08-597.png
>
>
> Attempting to open a multi-file dataset and write a re-partitioned version of it fails as it seems there is an attempt to collect data into memory first. This happens both for wide and long data.
> Steps to reproduce the issue:
> 1. Create a large dataset (100k columns, 300k rows) and write it to disk and create 20 copies of it. Each file will have a footprint of roughly 7.5GB. 
> {code:r}
> library(arrow)
> library(dplyr)
> library(fs)
> rows <- 300000
> cols <- 100000
> partitions <- 20
> wide_df <- as.data.frame(
>   matrix(
>     sample(1:32767, rows * cols / partitions, replace = TRUE), 
>     ncol = cols)
> )
> schem <- sapply(colnames(wide_df), function(nm) {int16()})
> schem <- do.call(schema, schem)
> wide_tab <- Table$create(wide_df, schema = schem)
> write_parquet(wide_tab, "~/Documents/arrow_playground/wide.parquet")
> fs::dir_create("~/Documents/arrow_playground/wide_ds")
> for (i in seq_len(partitions)) {
>   file.copy("~/Documents/arrow_playground/wide.parquet", 
>             glue::glue("~/Documents/arrow_playground/wide_ds/wide-{i-1}.parquet"))
> }
> ds_wide <- open_dataset("~/Documents/arrow_playground/wide_ds/")
> {code}
> All the following steps fail:
> 2. Creating and writing a partitioned version of {{{}ds_wide{}}}.
> {code:r}
>   ds_wide %>%
>     mutate(grouper = round(V1 / 1024)) %>%
>     write_dataset("~/Documents/arrow_playground/partitioned", 
>                    partitioning = "grouper",
>                    format = "parquet")
> {code}
> 3. Writing a non-partitioned dataset:
> {code:r}
>   ds_wide %>%
>     write_dataset("~/Documents/arrow_playground/partitioned", 
>                   format = "parquet")
> {code}
> 4. Creating the partitioning variable first and then attempting to write:
> {code:r}
>   ds2 <- ds_wide %>% 
>     mutate(grouper = round(V1 / 1024))
>   ds2 %>% 
>     write_dataset("~/Documents/arrow_playground/partitioned", 
>                   partitioning = "grouper", 
>                   format = "parquet")  
> {code}
> 5. Attempting to write to csv:
> {code:r}
> ds_wide %>% 
>   write_dataset("~/Documents/arrow_playground/csv_writing/test.csv",
>                 format = "csv")
> {code}
> None of the failures seem to originate in R code and they all result in a similar behaviour: the R sessions consume increasing amounts of RAM until they crash.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)