You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Dewey Dunnington (Jira)" <ji...@apache.org> on 2021/11/04 13:41:00 UTC
[jira] [Comment Edited] (ARROW-14575) [R] Allow functions with
{{pkg::}} prefixes
[ https://issues.apache.org/jira/browse/ARROW-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17438279#comment-17438279 ]
Dewey Dunnington edited comment on ARROW-14575 at 11/4/21, 1:40 PM:
--------------------------------------------------------------------
It might be nice for clarity to register the package name with the function (i.e., `arrow_register_translation("year", function(...) {}, package = "lubridate")` instead of `nse_funcs$year <- function(...) {}`.
Erroring out could look like this (only slightly better than the current error, though).
{noformat}
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- InMemoryDataset$create(nycflights13::flights)
funs <- arrow:::.cache$functions
funs[["::"]] <- function(lhs, rhs) {
arrow:::arrow_not_supported("`::`")
}
assign("functions", funs, envir = arrow:::.cache)
ds %>%
mutate(year = lubridate::year(time_hour)) %>%
collect()
#> Warning: In lubridate::year(time_hour), `::` not supported by Arrow; pulling
#> data into R
#> # A tibble: 336,776 × 19{noformat}
A slightly different definition of `::` could ignore it with a warning, although this is possibly a bit too magical.
{noformat}
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- InMemoryDataset$create(nycflights13::flights)
funs <- arrow:::.cache$functions
funs[["::"]] <- function(lhs, rhs) {
# already evaluated to an Expression?
rhs <- rhs$field_name
warning(paste0("Replacing `", as.character(substitute(lhs)), "::", rhs, "` with `", rhs, "`"))
get(rhs, envir = parent.frame(), mode = "function")
}
assign("functions", funs, envir = arrow:::.cache)ds %>%
mutate(year = lubridate::year(time_hour)) %>%
collect()
#> Warning in lubridate::year: Replacing `lubridate::year` with `year`
#> # A tibble: 336,776 × 19
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
#> <int> <int> <int> <int> <int> <dbl> <int> <int>
#> 1 2013 1 1 517 515 2 830 819
#> 2 2013 1 1 533 529 4 850 830
#> 3 2013 1 1 542 540 2 923 850
#> 4 2013 1 1 544 545 -1 1004 1022
#> 5 2013 1 1 554 600 -6 812 837
#> 6 2013 1 1 554 558 -4 740 728
#> 7 2013 1 1 555 600 -5 913 854
#> 8 2013 1 1 557 600 -3 709 723
#> 9 2013 1 1 557 600 -3 838 846
#> 10 2013 1 1 558 600 -2 753 745
#> # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>,
#> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
#> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>{noformat}
was (Author: paleolimbot):
It might be nice for clarity to register the package name with the function (i.e., `arrow_register_translation("year", function(...) {}, package = "lubridate")` instead of `nse_funcs$year <- function(...) {}`.
Erroring out could look like this (only slightly better than the current error, though).
{noformat}
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- InMemoryDataset$create(nycflights13::flights)funs <- arrow:::.cache$functions
funs[["::"]] <- function(lhs, rhs) {
arrow:::arrow_not_supported("`::`")
}
assign("functions", funs, envir = arrow:::.cache)
ds %>%
mutate(year = lubridate::year(time_hour)) %>%
collect()
#> Warning: In lubridate::year(time_hour), `::` not supported by Arrow; pulling
#> data into R
#> # A tibble: 336,776 × 19{noformat}
A slightly different definition of `::` could ignore it with a warning, although this is possibly a bit too magical.
{noformat}
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- InMemoryDataset$create(nycflights13::flights)
funs <- arrow:::.cache$functions
funs[["::"]] <- function(lhs, rhs) {
# already evaluated to an Expression?
rhs <- rhs$field_name
warning(paste0("Replacing `", as.character(substitute(lhs)), "::", rhs, "` with `", rhs, "`"))
get(rhs, envir = parent.frame(), mode = "function")
}
assign("functions", funs, envir = arrow:::.cache)ds %>%
mutate(year = lubridate::year(time_hour)) %>%
collect()
#> Warning in lubridate::year: Replacing `lubridate::year` with `year`
#> # A tibble: 336,776 × 19
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
#> <int> <int> <int> <int> <int> <dbl> <int> <int>
#> 1 2013 1 1 517 515 2 830 819
#> 2 2013 1 1 533 529 4 850 830
#> 3 2013 1 1 542 540 2 923 850
#> 4 2013 1 1 544 545 -1 1004 1022
#> 5 2013 1 1 554 600 -6 812 837
#> 6 2013 1 1 554 558 -4 740 728
#> 7 2013 1 1 555 600 -5 913 854
#> 8 2013 1 1 557 600 -3 709 723
#> 9 2013 1 1 557 600 -3 838 846
#> 10 2013 1 1 558 600 -2 753 745
#> # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>,
#> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
#> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>{noformat}
> [R] Allow functions with {{pkg::}} prefixes
> -------------------------------------------
>
> Key: ARROW-14575
> URL: https://issues.apache.org/jira/browse/ARROW-14575
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: Jonathan Keane
> Assignee: Dewey Dunnington
> Priority: Major
>
> Currently we implement a number of functions from packages like {lubridate} which work well when called without namespacing (e.g. {{year()}}), however if someone calls {{lubridate::year()}} we get a not-implemented method (e.g. {{Warning: Expression lubridate::year(time_hour) not supported in Arrow}}). Is it possible for us to look and see if we have an arrow function that matches the function itself.
> {code:r}
> library(arrow, warn.conflicts = FALSE)
> library(dplyr, warn.conflicts = FALSE)
> ds <- InMemoryDataset$create(nycflights13::flights)
> ds %>%
> mutate(year = lubridate::year(time_hour)) %>%
> collect()
> #> Warning: Expression lubridate::year(time_hour) not supported in Arrow; pulling
> #> data into R
> #> # A tibble: 336,776 × 19
> #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
> #> <dbl> <int> <int> <int> <int> <dbl> <int> <int>
> #> 1 2013 1 1 517 515 2 830 819
> #> 2 2013 1 1 533 529 4 850 830
> #> 3 2013 1 1 542 540 2 923 850
> #> 4 2013 1 1 544 545 -1 1004 1022
> #> 5 2013 1 1 554 600 -6 812 837
> #> 6 2013 1 1 554 558 -4 740 728
> #> 7 2013 1 1 555 600 -5 913 854
> #> 8 2013 1 1 557 600 -3 709 723
> #> 9 2013 1 1 557 600 -3 838 846
> #> 10 2013 1 1 558 600 -2 753 745
> #> # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>,
> #> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
> #> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
> ds %>%
> mutate(year = year(time_hour)) %>%
> collect()
> #> # A tibble: 336,776 × 19
> #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
> #> <int> <int> <int> <int> <int> <dbl> <int> <int>
> #> 1 2013 1 1 517 515 2 830 819
> #> 2 2013 1 1 533 529 4 850 830
> #> 3 2013 1 1 542 540 2 923 850
> #> 4 2013 1 1 544 545 -1 1004 1022
> #> 5 2013 1 1 554 600 -6 812 837
> #> 6 2013 1 1 554 558 -4 740 728
> #> 7 2013 1 1 555 600 -5 913 854
> #> 8 2013 1 1 557 600 -3 709 723
> #> 9 2013 1 1 557 600 -3 838 846
> #> 10 2013 1 1 558 600 -2 753 745
> #> # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>,
> #> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
> #> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)