Re: [I] Add option to record file path in `open_dataset` and `open_csv_dataset` [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #38036:
URL: https://github.com/apache/arrow/issues/38036#issuecomment-1748566132

   There's a function `add_filename()` which might be able to help you, e.g.
   
   ``` r
   library(arrow)
   #> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
   #> 
   #> Attaching package: 'arrow'
   #> The following object is masked from 'package:utils':
   #> 
   #>     timestamp
   library(dplyr)
   #> 
   #> Attaching package: 'dplyr'
   #> The following objects are masked from 'package:stats':
   #> 
   #>     filter, lag
   #> The following objects are masked from 'package:base':
   #> 
   #>     intersect, setdiff, setequal, union
   
   tf <- tempfile()
   dir.create(tf)
   write_dataset(group_by(mtcars, am), tf, format = "csv")
   
   list.files(tf, recursive = TRUE)
   #> [1] "am=0/part-0.csv" "am=1/part-0.csv"
   open_dataset(tf, format = "csv") %>%
     mutate(filename = add_filename()) %>%
     collect()
   #> # A tibble: 32 × 12
   #>      mpg   cyl  disp    hp  drat    wt  qsec    vs  gear  carb    am filename   
   #>    <dbl> <int> <dbl> <int> <dbl> <dbl> <dbl> <int> <int> <int> <int> <chr>      
   #>  1  21.4     6  258    110  3.08  3.22  19.4     1     3     1     0 /tmp/Rtmpf…
   #>  2  18.7     8  360    175  3.15  3.44  17.0     0     3     2     0 /tmp/Rtmpf…
   #>  3  18.1     6  225    105  2.76  3.46  20.2     1     3     1     0 /tmp/Rtmpf…
   #>  4  14.3     8  360    245  3.21  3.57  15.8     0     3     4     0 /tmp/Rtmpf…
   #>  5  24.4     4  147.    62  3.69  3.19  20       1     4     2     0 /tmp/Rtmpf…
   #>  6  22.8     4  141.    95  3.92  3.15  22.9     1     4     2     0 /tmp/Rtmpf…
   #>  7  19.2     6  168.   123  3.92  3.44  18.3     1     4     4     0 /tmp/Rtmpf…
   #>  8  17.8     6  168.   123  3.92  3.44  18.9     1     4     4     0 /tmp/Rtmpf…
   #>  9  16.4     8  276.   180  3.07  4.07  17.4     0     3     3     0 /tmp/Rtmpf…
   #> 10  17.3     8  276.   180  3.07  3.73  17.6     0     3     3     0 /tmp/Rtmpf…
   #> # ℹ 22 more rows
   ```
   
   <sup>Created on 2023-10-05 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org