You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Dewey Dunnington (Jira)" <ji...@apache.org> on 2022/04/11 15:05:00 UTC
[jira] [Commented] (ARROW-16144) [R] Write compressed data streams (particularly over S3)
[ https://issues.apache.org/jira/browse/ARROW-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520635#comment-17520635 ]
Dewey Dunnington commented on ARROW-16144:
------------------------------------------
I'm fairly sure this is implemented...is this the kind of behaviour you were looking for?
{code:R}
library(arrow, warn.conflicts = FALSE)
tf <- tempfile(fileext = ".gz")
write_csv_arrow(mtcars, tf)
readr::read_csv(gzfile(tf))
#> Rows: 32 Columns: 11
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> dbl (11): mpg, cyl, disp, hp, drat, wt, qsec, vs, am, gear, carb
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 32 × 11
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4
#> 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4
#> 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1
#> 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1
#> 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2
#> 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1
#> 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4
#> 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2
#> 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2
#> 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4
#> # … with 22 more rows
{code}
> [R] Write compressed data streams (particularly over S3)
> --------------------------------------------------------
>
> Key: ARROW-16144
> URL: https://issues.apache.org/jira/browse/ARROW-16144
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
> Affects Versions: 7.0.0
> Reporter: Carl Boettiger
> Priority: Major
>
> The python bindings have `CompressedOutputStream`, but I don't see how we can do this on the R side (e.g. with `write_csv_arrow()`). It would be wonderful if we could both read and write compressed streams, particularly for CSV and particularly for remote filesystems, where this can provide considerable performance improvements.
> (For comparison, readr will write a compressed stream automatically based on the extension for the given filename, e.g. `readr::write_csv(data, "file.csv.gz")` or `write_csv("data.file.xz")` )
--
This message was sent by Atlassian Jira
(v8.20.1#820001)