[jira] [Created] (ARROW-11582) write_dataset fails unexpectedly

["Carl Boettiger (Jira)" <ji...@apache.org>]

Carl Boettiger created ARROW-11582:
--------------------------------------

             Summary: write_dataset fails unexpectedly
                 Key: ARROW-11582
                 URL: https://issues.apache.org/jira/browse/ARROW-11582
             Project: Apache Arrow
          Issue Type: Bug
          Components: R
    Affects Versions: 2.0.0
         Environment: R 4.0.3, Ubuntu 20.04.

sessionInfo():



> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04 LTS

Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C              LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] rstudioapi_0.13   magrittr_2.0.1    hms_0.5.3         tidyselect_1.1.0  bit_4.0.4         R6_2.5.0          rlang_0.4.9      
 [8] dplyr_1.0.2       tools_4.0.3       R.oo_1.24.0       arrow_2.0.0       DBI_1.1.0         ellipsis_0.3.1    bit64_4.0.5      
[15] assertthat_0.2.1  tibble_3.0.4      lifecycle_0.2.0   crayon_1.3.4      readr_1.4.0       purrr_0.3.4       arkdb_0.0.8      
[22] duckdb_0.2.4      fs_1.5.0          vctrs_0.3.5       R.utils_2.10.1    curl_4.3          glue_1.4.2        compiler_4.0.3   
[29] pillar_1.4.7      generics_0.1.0    R.methodsS3_1.8.1 pkgconfig_2.0.3  
            Reporter: Carl Boettiger


 

I'd like to use the R package interface to access data distributed in a tab-separated text file that is much larger than available RAM.  I understand that in principle this is possible using `open_datatset()` in text mode and then streaming data out to parquet via `write_dataset()`, but this strategy fails even on small text files with an unexpected error:

Here's a minimal reproducible example.

 
fs::dir_create("import_dir")
readr::write_tsv(mtcars, "import_dir/mtcars.tsv")
ds <- arrow::open_dataset("import_dir", format="text", delim="\t")
arrow::write_dataset(ds, "parquet_dir")
The error I get occurs only on the last line (`write_dataset()`), saying: 
Error in options$update(...) : attempt to apply non-function
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)