You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ablack3 (via GitHub)" <gi...@apache.org> on 2023/09/17 19:50:39 UTC
[GitHub] [arrow] ablack3 opened a new issue, #37760: existing_data_behavior = "overwrite" does not work as expected in arrow::write_dataset
ablack3 opened a new issue, #37760:
URL: https://github.com/apache/arrow/issues/37760
### Describe the bug, including details regarding any error messages, version, and platform.
It seems that the default for existing_data_behavior='overwrite' does not actually overwrite existing data. @jonkeane
``` r
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
path <- file.path(tempdir(), "mtcars-data")
mtcars %>%
group_by(cyl) %>%
arrow::write_dataset(path)
list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8"
fs::file_size(path)
#> 160
mtcars %>%
group_by(vs) %>%
arrow::write_dataset(path, existing_data_behavior = "overwrite")
list.dirs(path, full.names = F)
#> [1] "" "cyl=4" "cyl=6" "cyl=8" "vs=0" "vs=1"
fs::file_size(path)
#> 224
sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5.2
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> time zone: America/Chicago
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] dplyr_1.1.3
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.6.3 cli_3.6.1 knitr_1.44 rlang_1.1.1
#> [5] xfun_0.40 purrr_1.0.2 generics_0.1.3 assertthat_0.2.1
#> [9] bit_4.0.5 glue_1.6.2 htmltools_0.5.6 fansi_1.0.4
#> [13] rmarkdown_2.24 evaluate_0.21 tibble_3.2.1 fastmap_1.1.1
#> [17] yaml_2.3.7 lifecycle_1.0.3 compiler_4.3.1 fs_1.6.3
#> [21] pkgconfig_2.0.3 rstudioapi_0.15.0 digest_0.6.33 R6_2.5.1
#> [25] reprex_2.0.2 tidyselect_1.2.0 utf8_1.2.3 pillar_1.9.0
#> [29] magrittr_2.0.3 bit64_4.0.5 tools_4.3.1 withr_2.5.0
#> [33] arrow_13.0.0
```
<sup>Created on 2023-09-17 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup>
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on issue #37760: [R] existing_data_behavior = "overwrite" does not work as expected in arrow::write_dataset
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on issue #37760:
URL: https://github.com/apache/arrow/issues/37760#issuecomment-1723920689
(sorry for the incorrect link up there, too many tabs open, apparently!)
Ah, I see what's going on here. In the overwrite mode we anticipated someone was overwriting a dataset with _the same partitioning_. If I remember correctly, the way that we delete files in overwrite mode, the C++ looks for the folders the partitions would create and then deletes the files inside of those. But if you write with a new partitioning, it doesn't look in other folders that might exist.
There are a few things we could do (and these aren't necessarily mutually exclusive):
* Update the documentation to explain this
* Add a check in either R or C++ to check if the data directory is empty or contains what looks like a dataset and either warn or remove things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org