[jira] [Commented] (ARROW-13472) [R] Redesign the UX for using the DuckDB engine

["Jonathan Keane (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388735#comment-17388735 ] 

Jonathan Keane commented on ARROW-13472:
----------------------------------------

Thanks for this and putting it into these words. I agree that the {{.engine}} argument while convenient, is awkward and departs from the experience of {{summarise}} and the rest of the other dplyr verbs like you mention. That was the major motivation in creating the {{to_duckdb}} function.

Though in that PR (this was in the original description + you can see it in the commit history, though both are relatively hidden now), I came to the conclusion that neither {{collect()}} nor {{compute()}} was a good match for what we were doing where a person could pass an Arrow object to duckdb (or other system) in the middle of a pipeline. The biggest reason being that both {{collect()}} and {{compute()}} imply that work is being done / executed at that point which for this it is not. I toyed with the idea of calling this {{alchemise()}} (or having a family like {{alchemise_to_duckdb()}} so that one knows what the function returns from the name + it is consistent), so that one could do something like the following:

{code:r}
ds <- InMemoryDataset$create(example_data)

ds %>%
  alchemize(to = "duckdb") %>%
  group_by(lgl) %>%
  summarise(mean_int = mean(int, na.rm = TRUE), mean_dbl = mean(dbl, na.rm = TRUE)) %>%
  alchemize(to = "arrow") %>%
  collect()
{code}

> [R] Redesign the UX for using the DuckDB engine
> -----------------------------------------------
>
>                 Key: ARROW-13472
>                 URL: https://issues.apache.org/jira/browse/ARROW-13472
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: R
>            Reporter: Ian Cook
>            Priority: Major
>             Fix For: 6.0.0
>
>
> ARROW-12688 added:
>  * A new function {{to_duckdb()}} which registers an Arrow Dataset with DuckDB and returns a dbplyr object that can be used in dplyr pipelines
>  * An {{.engine = "duckdb"}} option in the {{summarise()}} function which calls {{to_duckdb()}} inside {{summarise()}}
> At the moment, the latter is very convenient because {{summarise()}} is not yet natively supported for Arrow Datasets.
> However, this {{.engine = "duckdb"}} option is probably not such a great design for how users should interact with the arrow package in the longer term after native {{summarise()}} support is added. At that point, it will seem strange that this one particular dplyr verb has an {{.engine}} option while the others do not. Adding the option to all the other dplyr verbs also seems like a poor UX design.
> Consider whether we should ultimately have users choose whether to use the Arrow C++ engine or the DuckDB engine by passing an {{.engine}} argument to the {{collect()}} or {{compute()}} function, as [~jonkeane] suggested in these comments. {{collect()}} would return a tibble whereas {{compute()}} would return an Arrow Table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)