[jira] [Closed] (ARROW-10945) [Rust] [DataFusion] Allow User Defined Aggregates to return multiple values / structs

["Andrew Lamb (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Lamb closed ARROW-10945.
-------------------------------
    Resolution: Duplicate

Moved to https://github.com/apache/arrow-datafusion/issues/600

> [Rust] [DataFusion] Allow User Defined Aggregates to return multiple values / structs
> -------------------------------------------------------------------------------------
>
>                 Key: ARROW-10945
>                 URL: https://issues.apache.org/jira/browse/ARROW-10945
>             Project: Apache Arrow
>          Issue Type: New Feature
>            Reporter: Andrew Lamb
>            Priority: Major
>
> Usecase:
> I want to implement a user defined aggregate function that produces more than one column ( logical values)
> Specifically I am trying to implement the InfluxDB 'selector' functions `first`, `last`, `min`, and `max` as DataFusion aggregate functions.
> I can't use the built in aggregate functions in DataFusion as selector functions aren't exactly like normal aggregate functions -- they return both the actual aggregate value as well as a timestamp. In addition, `first` and `last` pick a row in the value column based on the value in the timestamp column.
> After some investigation, I realize I can't elegantly use the built in user defined aggregate framework in DataFusion either. As an example of what is going on here, let's take
> ```
> value | time
> ------+------
>   3   | 1000
>   2   | 2000
>   1   | 3000
> ```
> The result of `last(value)` should be be two columns `1 | 3000` -- however, modeling this as a DataFusion aggregate does not seem to be possible at this time.  Each aggregate function can return a single columnar value but we need to return 2 (the `.value` and `.time` fields).
> Ideally I was thinking that the UDF could produce a Struct (with named field `value` and `time`) but the evaluate function([code])(https://github.com/apache/arrow/blob/master/rust/datafusion/src/physical_plan/mod.rs#L238))returns a `ScalarValue` and at the moment they [don't have support for Structs](https://github.com/apache/arrow/blob/master/rust/datafusion/src/scalar.rs#L44)
> I suspect that we would also need to add support in DataFusion for selecting fields from structs
> See additional detail and context on https://github.com/influxdata/influxdb_iox/issues/448#issuecomment-744601824



--
This message was sent by Atlassian Jira
(v8.3.4#803005)