You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gstvg (via GitHub)" <gi...@apache.org> on 2025/11/25 04:33:53 UTC
[PR] [DRAFT] Add lambda support and array_transform udf [datafusion]
gstvg opened a new pull request, #18921:
URL: https://github.com/apache/datafusion/pull/18921
This PR adds support for lambdas with column capture and the array_transform scalar function which is used to test the lambda implementation. The changes are extensive, across various parts of the codebase, mostly tree traversals. This text aims to justify those changes, and show alternatives which may require less changes although not without trade-offs, so we can decide whether to move this forward or not, and if so, with what approach. For those who want to take a look at the code, don't waste time in the second commit as it is just adding a new field to a struct. [There's a build of the documentation of this branch available online](https://gstvg.github.io/datafusion/index.html?search=lambda)
Example array_transform usage:
```sql
array_transform([1, 2], v -> v*2)
[2, 4]
```
<details><summary>Lambda Logical and Physical Representation</summary>
```rust
pub struct Lambda {
pub params: Vec<String>, // in the example above, vec!["v"]
pub body: Box<Expr>, // in the example above, v*2
}
pub struct LambdaExpr {
params: Vec<String>, // in the example above, vec!["v"]
body: Arc<dyn PhysicalExpr>, // in the example above, v * 2
}
```
</details>
# Changes in ScalarUDF[Impl] to support lambdas
Since lambda parameters are defined by the UDF implementation, datafusion doesn't know the type, nullability nor its metadata. So, we add a method to ScalarUDF[Impl], where the implementation returns a Field for each parameter supported for each of its lambdas
<details ><summary>ScalarUDF[Impl] lambdas_parameters method</summary>
```rust
struct/trait ScalarUDF[Impl] {
/// Returns the parameters that any lambda supports
fn lambdas_parameters(
&self,
args: &[ValueOrLambdaParameter],
) -> Result<Vec<Option<Vec<Field>>>> {
Ok(vec![None; args.len()])
}
}
pub enum ValueOrLambdaParameter<'a> {
/// A columnar value with the given field
Value(FieldRef),
/// A lambda with the given parameters names and a flag indicating whether it captures any columns
Lambda(&'a [String], bool),
}
```
</details>
<details><summary>ArrayTransform lambdas_parameters implementation</summary>
```rust
impl ScalarUDFImpl for ArrayTransform {
fn lambdas_parameters(
&self,
args: &[ValueOrLambdaParameter],
) -> Result<Vec<Option<Vec<Field>>>> {
let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda(_, _)] =
args
else {
return exec_err!(
"{} expects a value followed by a lambda, got {:?}",
self.name(),
args
);
};
let (field, index_type) = match list.data_type() {
DataType::List(field) => (field, DataType::Int32),
DataType::LargeList(field) => (field, DataType::Int64),
DataType::FixedSizeList(field, _) => (field, DataType::Int32),
_ => return exec_err!("expected list, got {list}"),
};
// we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
// nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
// as datafusion will do that for us
let value = Field::new("value", field.data_type().clone(), field.is_nullable())
.with_metadata(field.metadata().clone());
let index = Field::new("index", index_type, false);
Ok(vec![None, Some(vec![value, index])])
}
}
```
</details>
<br>
In order for the UDF to be able to compute its return field, it usually needs to know the return field of its lambdas, which is the case for array_transform. Because lambdas may capture columns, to compute the return field of a lambda, it's necessary to know the Field of the captured columns, which is only available in the schema and it's not passed as argument to return_fields[_from_args]. To avoid changing that, we internally use the fields returned in the newly added method ScalarUDFImpl::lambdas_parameters paired with the schema to compute the return field of the lambdas, and pass them in ReturnFieldArgs.arg_fields. We also add a slice of bools indicating if the arg in the position `i` is a lambda or not. Finally, we add a helper method to_lambda_args which merges the data in arg_fields and lambdas into a vec of ValueOrField enums allowing convenient pattern matching instead of having to inspect both arg_fields and lambdas.
<br>
<details ><summary>ReturnFieldArgs changes</summary>
```rust
pub struct ReturnFieldArgs<'a> {
/// The data types of the arguments to the function
///
/// If argument `i` to the function is a lambda, it will be the return field of the
/// lambda expression when evaluated with the arguments returned from
/// ScalarUDFImpl::lambdas_parameters`
///
/// For example, with `array_transform([1], v -> v == 5)`
/// this field will be `[Field::new("", DataType::List(DataType::Int32), false), Field::new("", DataType::Boolean, false)]`
pub arg_fields: &'a [FieldRef],
... skipped fields
/// Is argument `i` to the function a lambda?
///
/// For example, with `array_transform([1], v -> v == 5)`
/// this field will be `[false, true]`
pub lambdas: &'a [bool],
}
/// A tagged Field indicating whether it correspond to a value or a lambda argument
#[derive(Debug)]
pub enum ValueOrLambdaField<'a> {
/// The Field of a ColumnarValue argument
Value(&'a FieldRef),
/// The Field of the return of the lambda body when evaluated with the parameters from ScalarUDF::lambda_parameters
Lambda(&'a FieldRef),
}
impl<'a> ReturnFieldArgs<'a> {
/// Based on self.lambdas, encodes self.arg_fields to tagged enums
/// indicating whether it correspond to a value or a lambda argument
pub fn to_lambda_args(&self) -> Vec<ValueOrLambdaField<'a>> {
std::iter::zip(self.arg_fields, self.lambdas)
.map(|(field, is_lambda)| {
if *is_lambda {
ValueOrLambdaField::Lambda(field)
} else {
ValueOrLambdaField::Value(field)
}
})
.collect()
}
}
```
</details>
<details><summary>ArrayTransform return_field_from_args implementation</summary>
```rust
impl ScalarUDFImpl for ArrayTransform {
fn return_field_from_args(
&self,
args: datafusion_expr::ReturnFieldArgs,
) -> Result<Arc<Field>> {
let args = args.to_lambda_args();
let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
take_function_args(self.name(), &args)?
else {
return exec_err!(
"{} expects a value followed by a lambda, got {:?}",
self.name(),
args
);
};
// lambda is the resulting field of executing the lambda body
// with the parameters returned in lambdas_parameters
let field = Arc::new(Field::new(
Field::LIST_FIELD_DEFAULT_NAME,
lambda.data_type().clone(),
lambda.is_nullable(),
));
let return_type = match list.data_type() {
DataType::List(_) => DataType::List(field),
DataType::LargeList(_) => DataType::LargeList(field),
DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
_ => exec_err!("expected list, got {list}")?,
};
Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
}
}
```
</details>
<br>
For execution, we add the lambdas fields to ScalarFunctionArgs and the helper method to_lambda_args, similar to the changes in ReturnFieldArgs and its to_lambda_args:
<br>
<details><summary>ScalarFunctionArgs changes</summary>
```rust
pub struct ScalarFunctionArgs {
/// The evaluated arguments to the function
/// If it's a lambda, will be `ColumnarValue::Scalar(ScalarValue::Null)`
///
/// For example, with `array_transform([1], v -> v == 5)`
/// this field will be `[ColumnarValue::Scalar(ScalarValue::List([1])), ColumnarValue::Scalar(ScalarValue::Null)]`
pub args: Vec<ColumnarValue>,
/// Field associated with each arg, if it exists
pub arg_fields: Vec<FieldRef>,
....
/// The lambdas passed to the function
/// If it's not a lambda it will be `None`
///
/// For example, with `array_transform([1], v -> v == 5)`
/// this field will be `[None, Some(...)]`
pub lambdas: Option<Vec<Option<ScalarFunctionLambdaArg>>>,
}
/// A lambda argument to a ScalarFunction
#[derive(Clone, Debug)]
pub struct ScalarFunctionLambdaArg {
/// The parameters defined in this lambda
///
/// For example, for `array_transform([2], v -> -v)`,
/// this will be `vec![Field::new("v", DataType::Int32, true)]`
pub params: Vec<FieldRef>,
/// The body of the lambda
///
/// For example, for `array_transform([2], v -> -v)`,
/// this will be the physical expression of `-v`
pub body: Arc<dyn PhysicalExpr>,
/// A RecordBatch containing at least the captured columns inside this lambda body, if any
/// Note that it may contain additional, unspecified columns, but that's an implementation detail
///
/// For example, for `array_transform([2], v -> v + a + b)`,
/// this will be a `RecordBatch` with at least columns `a` and `b`
pub captures: Option<RecordBatch>,
}
impl ScalarFunctionArgs {
pub fn to_lambda_args(&self) -> Vec<ValueOrLambda<'_>> {
match &self.lambdas {
Some(lambdas) => std::iter::zip(&self.args, lambdas)
.map(|(arg, lambda)| match lambda {
Some(lambda) => ValueOrLambda::Lambda(lambda),
None => ValueOrLambda::Value(arg),
})
.collect(),
None => self.args.iter().map(ValueOrLambda::Value).collect(),
}
}
}
/// An argument to a higher-order scalar function
pub enum ValueOrLambda<'a> {
Value(&'a ColumnarValue),
Lambda(&'a ScalarFunctionLambdaArg),
}
```
</details>
<details><summary>ArrayTransform invoke_with_args implementation</summary>
```rust
impl ScalarUDFImpl for ArrayTransform {
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
// args.lambda_args allows the convenient match below, instead of inspecting both args.args and args.lambdas
let lambda_args = args.to_lambda_args();
let [list_value, lambda] = take_function_args(self.name(), &lambda_args)?;
let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
(list_value, lambda)
else {
return exec_err!(
"{} expects a value followed by a lambda, got {:?}",
self.name(),
&lambda_args
);
};
let list_array = list_value.to_array(args.number_rows)?;
// if any column got captured, we need to adjust it to the values arrays,
// duplicating values of list with multiple values and removing values of empty lists
// list_indices is not cheap so is important to avoid it when no column is captured
let adjusted_captures = lambda
.captures
.as_ref()
.map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
.transpose()?;
// use closures and merge_captures_with_lazy_args so that it calls only the needed ones based on the number of arguments
// avoiding unnecessary computations
let values_param = || Ok(Arc::clone(list_values(&list_array)?));
let indices_param = || elements_indices(&list_array);
// the order of the merged schema is an unspecified implementation detail that may change in the future,
// using this function is the correct way to merge as it return the correct ordering and will change in sync
// the implementation without the need for fixes. It also computes only the parameters requested
let lambda_batch = merge_captures_with_lazy_args(
adjusted_captures.as_ref(),
&lambda.params, // ScalarUDF already merged the fields returned in lambdas_parameters with the parameters names defined in the lambda, so we don't need to
&[&values_param, &indices_param],
)?;
// call the transforming expression with the record batch composed of the list values merged with captured columns
let transformed_values = lambda
.body
.evaluate(&lambda_batch)?
.into_array(lambda_batch.num_rows())?;
let field = match args.return_field.data_type() {
DataType::List(field)
| DataType::LargeList(field)
| DataType::FixedSizeList(field, _) => Arc::clone(field),
_ => {
return exec_err!(
"{} expected ScalarFunctionArgs.return_field to be a list, got {}",
self.name(),
args.return_field
)
}
};
let transformed_list = match list_array.data_type() {
DataType::List(_) => {
let list = list_array.as_list();
Arc::new(ListArray::new(
field,
list.offsets().clone(),
transformed_values,
list.nulls().cloned(),
)) as ArrayRef
}
DataType::LargeList(_) => {
let large_list = list_array.as_list();
Arc::new(LargeListArray::new(
field,
large_list.offsets().clone(),
transformed_values,
large_list.nulls().cloned(),
))
}
DataType::FixedSizeList(_, value_length) => {
Arc::new(FixedSizeListArray::new(
field,
*value_length,
transformed_values,
list_array.as_fixed_size_list().nulls().cloned(),
))
}
other => exec_err!("expected list, got {other}")?,
};
Ok(ColumnarValue::Array(transformed_list))
}
}
```
</details>
<br>
# Changes in tree traversals
Using this query as an example:
```sql
create table t as select 1 as a, [[2, 3]] as b, 1 as c;
select a, b, c, array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) from t;
a | b | c | array_transform(b, (b, i) -> array_transform(b, b -> b + c + i))
1, [[2, 3]], 1, [[4, 5]]
```
Detailing the identifiers in the query:
```sql
definition of lambda first parameter, arbitrally named "b", the element of the array being
transformed. In the example, a List(Int32) column of [2, 3] . shadows t.b
^
|
| definition of lambda second <--+
| parameter arbitrally named "i" |
| the 1-based index of the element | definition of the only parameter of the lambda,
| being transformed: in the | arbitrally named "b", the element of the array
| example, a Int32 column of "1" | being transformed. Shadows the parameter "b"
| | from outer lambda. The index parameter is omitted.
| | In the example, a Int32 column with values "2, 3"
+-------------------------------+ | ^
| | |
| | |
select a, b, c, array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) from t;
| | | | | | | |
| | | v | | | |
| | | column "b" from table "t", t.b being passed | | | |
| | | as argument to outer array_transform | | | |
| | v | | | |
| | projection of column "c" from table "t", t.c | | | |
| | | | | |
| v | | | |
| projection of column "b" from table "t", t.b | | | |
| | | | |
v | | | |
projection of column "a" from table "t", t.a | | | |
| | | |
v | | |
parameter "b" from outer lambda being passed | | |
as argument to the inner array_transform | | |
v | |
reference to parameter "b" from inner lambda | |
| |
v |
reference to column "c" captured from table "t", t.c |
|
v
reference to parameter "i" captured from outer lambda
```
Note that:
1: lambdas may be nested
2: lambdas may capture columns from the outer scope, be it from the input plan or from another, outer lambdas.
3: lambdas introduces parameters that shadows columns from the outer scope
4: lambdas may support multiple parameters, and it is possible to omit the trailing ones that aren't used. Omitting unnecessary parameters positioned before an used parameter is currently not supported and may incur unnecessary computations
# Representing columns referring lambda parameters while being able to differentiate them from regular columns in Expr tree traversals
Because they are identical to regular columns, it is intuitive to use the same logical and physical expression to represent columns referring to lambdas parameters. However, the existing tree traversals were made without taking into account that a column may refer to a lambda parameter, and not a column from the input plan, and so they would behave erratically. In the example query, projection pushdown would try to push the lambda parameter "i", which won't exist in table "t".
Another example:
```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
let mut used_columns = HashSet::new();
expr.apply(|expr| {
if let Some(col) = expr.as_any().downcast_ref::<Column>() {
// if this is a lambda column, code below will break
used_columns.insert(col.index());
}
Ok(TreeNodeRecursion::Continue)
});
...
}
```
Therefore, we either make available a way to differentiate them, or use two different expressions:
## Option 1. Use the same Column expression, differentiate with a new set of TreeNode methods, *_with_lambdas_params
This PR uses the existing column expression, always unqualified, and adds a new set of [TreeNode-like methods](https://gstvg.github.io/datafusion/index.html?search=with_lambdas_params) on expressions that starts traversals with an empty HashSet<String>, and while traversing the expr tree, when finding a lambda, clone the set and adds the lambda parameters to it, and pass it to the visiting/transforming closure so that it can differentiate the columns
```rust
impl Expr {
pub fn transform_with_lambdas_params<
F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
>(
self,
mut f: F,
) -> Result<Transformed<Self>> {}
}
```
<details><summary>Expr tree traversal with lambdas_params. This query is a modified version of the example query where the inner lambda second parameter is used</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, (b, j) -> b + i + j)) ╷
╷ ╷
╷ ╷
╷ lambdas_params = {} ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, (b, j) -> b + i + j) ╷
╷ ╷
╷ ╷
╷ lambdas_params = { b, i } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, j) -> b + i + j) ╷
╷ ╷
╷ ╷
╷ lambdas_params = { b, i } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, j) -> b + i + j ╷
╷ ╷
╷ ╷
╷ lambdas_params = { b, i, j } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
How minimize_join_filter would looks like:
```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
let mut used_columns = HashSet::new();
expr.apply_with_lambdas_params(|expr, lambdas_params| {
if let Some(col) = expr.as_any().downcast_ref::<Column>() {
// dont include lambdas parameters
if !lambdas_params.contains(col.name()) {
used_columns.insert(col.index());
}
}
Ok(TreeNodeRecursion::Continue)
})
...
}
```
I started with this option without an idea of how big it would be. It requires using the new TreeNode methods and checking the column in 30+ tree traversals, and so will downstream too. I choose to keep it in this PR so that we only choose/discard this knowing how big it is.
## Option 2. New Expr LambdaColumn
Create a new Expr, LambdaColumn, which doesn't require new TreeNode methods, but requires that expr_api users use this new expr when applicable. It requires a fix in expr_simplifier for expressions with a lambda column, and may require similar work in downstream.
```rust
struct LambdaColumn {
name: String,
spans: Spans,
}
struct LambdaColumn {
name: String,
index: usize,
}
```
Existing code inalterated
```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
let mut used_columns = HashSet::new();
expr.apply(|expr| {
if let Some(col) = expr.as_any().downcast_ref::<Column>() {
//no need to check because Column is never a LambdaColumn
used_columns.insert(col.index());
}
Ok(TreeNodeRecursion::Continue)
})
...
}
```
## Option 3. Add is_lambda_parameter boolean field to Column
Add is_lambda_parameter to the existing column expression. It won't require new TreeNode methods, but still requires checking the field everywhere that new TreeNode methods are currently used in this PR
```rust
//logical
struct Column {
pub relation: Option<TableReference>,
pub name: String,
pub spans: Spans,
pub is_lambda_parameter: bool,
}
//physical
struct Column {
name: String,
index: usize,
is_lambda_parameter: bool,
}
```
How minimize_join_filter would look like:
```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
let mut used_columns = HashSet::new();
expr.apply(|expr| {
if let Some(col) = expr.as_any().downcast_ref::<Column>() {
// dont include lambdas parameters
if !col.is_lambda_parameter {
used_columns.insert(col.index());
}
}
Ok(TreeNodeRecursion::Continue)
})
...
}
```
### Comparison between options:
| | Expr::Column/ColumnExpr | Expr::LambdaColumn/LambdaColumnExpr | is_lambda_parameter flag in Column |
|-------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| Internal code change | New TreeNode methods, use them where applicable | Add new expr. Use the correct expr while parsing sql(instrumentation already exists to skip normalization of lambdas parameters in sql parsing) | Check the field where applicable. Set the field while parsing sql |
| Downstream code change | If lambda support is desired, use the new TreeNode where applicable. Otherwise none | None | If lambda support is desired, check the field where applicable Otherwise none |
| Create a new Expr | No | Yes | No |
| Requires new TreeNode methods | Yes, _with_lambdas_params for both logical and physical expressions | No | No |
| Inhibits existing optimizations for exprs with columns of lambdas parameters | No | expr_simplifier, fixable. But may happen in downstream too | No |
| expr_api users must reason about | No | Yes, use the correct expression type | Yes, set the flag |
| Two almost identical Expressions which major difference is the place in the expr tree that they exist | No | Yes | No |
| Data from the tree leaks into the column node | No | No | Yes, is_lambda_parameter is a information not about the column node itself, but about its place in the tree |
Ultimately, I don't have an inclination for any option and I believe it's a decision up to maintainers and those with more contact with downstream users, who have more idea of what option is easier to use. I think that the most laborious being already implemented puts us in a good position to make a choice, and would be easy to change to another option.
Continue in the comment below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2898017709
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
Thanks @LiaCastaneda for the info and @benbellick for the release
> I wonder if the expressions can actually be mapped 1:1
I did a quick check and I believe it's possible, yes.
`LambdaFunction` produces a regular substrait `ScalarFunction`, and to consume we just check whether any `LambdaUDF` exists with the given function name and/or if any arg is a lambda.
To produce a substrait `Lambda` with the field `parameters` struct where each field corresponds to a parameter, we use the return of `LambdaUDF::lambdas_parameters`, and for consuming, we can use default parameters names like `p1`, `p2` etc
To produce a `FieldReference`/`LambdaParameterReference` with a correct `steps_out` and `direct_reference.struct_field.field`, we keep a `HashMap<String, (usize, usize)>` of `lambda_parameter_name => (step_outs, struct_field_index)`, that get's updated for every `Lambda` to be produced: `step_outs` is reset to 0 for shadowed parameters, others incremented by one, and `struct_field_index` set according to it's position within the parameters of the lambda it originates from. Consuming into `LambdaVariable` with it's field resolved is similar to SQL planning: keep a `HashMap<usize, FieldRef>` of `direct_reference.struct_field.field => field` that get's updated for every `LambdaUDF` by the return of it's `lambdas_parameters` method
I'll address your another comments and then implement substrait support on another branch to make sure is viable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3017104135
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,104 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ ///
+ /// For example, `array_transform([1,2,3], v -> v+1)` would be equivalent to:
+ ///
+ /// ```text
+ /// LambdaFunction(array_transform)
+ /// ├── args[0]: Literal([1,2,3])
+ /// └── args[1]: Lambda
+ /// ├── params: ["v"]
+ /// └── body: BinaryExpr(+)
+ /// ├── LambdaVariable("v")
+ /// └── Literal(1)
+ /// ```
+ LambdaFunction(LambdaFunction),
Review Comment:
I did a quick search to see what other engines support. None support function references and instead require a wrapper lambda (e.g. `x -> function(x)`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2985104418
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
You said before that you have some list cleanup helpers, if you think that we should use them instead of the current ones just share them with me, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4261592099
@comphead @rluvaton I think it's great, new PR created at #21679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3085194423
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
> Maybe in the future we can add a null_safe method to PhysicalExpr and only cleanup if any expr of the lambda body is non null safe?
Sorry I missed your answer! Yeah, I think we can leave this for when someone actually needs it -- I don't think we should add APIs if there is no real use case yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3003053330
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
Hmm actually since capture support has been scraped for now we can determine by index without having to care about that and leave it for the capture PR https://github.com/apache/datafusion/pull/18921/commits/93e66f7c618cfab4acb8f85588d1727d6b4b7abf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3930657951
Thanks @gstvg for driving this. I missed this PR and actually was advocating for lambda support in DataFusion roadmap. Let me explore this approach, in high level IMO it makes sense to introduce LambdaUDF trait so taking as example `SELECT list_filter([1,2,3,4], x -> x % 2 = 0);` it could look like
```
FunctionExpression
name: list_filter
children:
1. ListExpression [1,2,3,4]
2. LambdaExpression
parameters: ["x"]
body:
ComparisonExpression (=)
left:
ArithmeticExpression (%)
left: ColumnRef("x")
right: Constant(2)
right:
Constant(0)
```
Then we likely need to Bind `x` to know what is it and finally rewrite it into something computable, perhaps using UDF of (Int) => Boolean and provide a bitmap as the result of valid indices and then extract valid indices from array. It might preserve SIMD execution.
It is great to see this effort is moving
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [DRAFT] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3573747241
# Traversing Expr trees with a schema that include lambdas parameters
The parameters of a lambda aren't present in the schema of the plan they belong to. During tree traversals that use a schema to check expressions datatype, nullability and metadata, there must be a way to access a schema which includes those parameters.
<details><summary>Expr tree traversal with wrong schema</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
╷ !! missing "i", incorrect "b" type !! ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
╷ !! missing "i", incorrect "b" type !! ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
╷ !! missing "i", incorrect "b" type !! ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
## Option 1. Per-lambda schema with a new set of TreeNode methods: *_with_schema
Once again, this PR adds another set of TreeNode-like methods on [logical](https://gstvg.github.io/datafusion/index.html?search=expr%3A%3Awith_schema) and [physical](https://gstvg.github.io/datafusion/physical_expr/trait.PhysicalExprExt.html) expressions, that while traversing expression trees, when they find a ScalarUDF that contains a lambda on its arguments, uses ScalarUDF::lambdas_parameters to create a schema adjusted for each of its arguments, and pass it as an argument to the visiting/transforming function.
```rust
impl Expr {
pub fn transform_with_schema<
F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
>(
self,
schema: &DFSchema,
f: F,
) -> Result<Transformed<Self>> {}
}
```
Example usage:
```rust
pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
let mut has_placeholder = false;
// Provide the schema as the first argument.
// Transforming closure receive an adjusted_schema as argument
self.transform_with_schema(schema, |mut expr, adjusted_schema| {
match &mut expr {
// Default to assuming the arguments are the same type
Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
// use adjusted_schema and not schema. Those expressions may contain
// columns referring to a lambda parameter, which Field would only be
// available in adjusted_schema and not in schema
rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
}
....
```
In order to add the lambda parameters to schema, we need to take into account DFSchema properties:
"Unqualified fields must be unique not only amongst themselves, but also must have a distinct name from any qualified field names"
Since lambdas parameters are always unqualified, they may conflict with columns of the outer schema, even though those being qualified. To fix this conflict, we can either:
1: Replace the existing column with the lambda parameter, in the same index of the vec of fields of the schema, in order to not change the index of columns to the right of it. That's the current approach in this PR
<details><summary>Expr tree traversal with adjusted schema, replacing conflicts</summary>
```sql
+------------------------------------------------------------------+
| array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) |
| |
| a = Int32 |
| b = List(List(Int32)) |
| c = Int32 |
+------------------------------------------------------------------+
|
|
v
+------------------------------------------------------------------+
| (b, i) -> array_transform(b, b -> b + c + i) |
| |
| a = Int32 |
| b = List(Int32) ! replaced ! |
| c = Int32 |
| i = Int32 |
+------------------------------------------------------------------+
|
|
v
+------------------------------------------------------------------+
| array_transform(b, b -> b + c + i) |
| |
| a = Int32 |
| b = List(Int32) ! replaced ! |
| c = Int32 |
| i = Int32 |
+------------------------------------------------------------------+
|
|
v
+------------------------------------------------------------------+
| b -> b + c + i |
| |
| a = Int32 |
| b = Int32 ! replaced ! |
| c = Int32 |
| i = Int32 |
+------------------------------------------------------------------+
```
</details>
<br>
<br>
2: Rename the shadowed column to an unique, non-conflicting name and add the lambda parameter to the end of the vec of fields of the schema. This option allows checking if a physical column refers to a lambda parameter by checking if its index is greater or equal than the number of fields of the outer schema. When this information is available, it eliminates the need to use the with_lambdas_params variations of TreeNode methods. It's trivial to change the PR to use this.
<details><summary>Expr tree traversal with adjusted schema, renaming conflicts</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ a = Int32 ╷
╷ b_shadowed1 = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ b = List(Int32) ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ a = Int32 ╷
╷ b_shadowed1 = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ b = List(Int32) ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ a = Int32 ╷
╷ b_shadowed1 = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ b_shadowed2 = List(Int32) ╷
╷ b = Int32 ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
<br>
<br>
<br>
Lambdas usually are evaluated with a different number of rows than that of the outer scope, as in the example, where array_transform is executed with one row, and its lambda with two rows, one for each element of the array. The UDF implementation is responsible for adjusting the captured columns with the number of rows of its parameters with whatever logic makes sense to it. For array_transform, its to copy the value of the captured column for each element of the arrays:
```
copied once a [1]------------------> a 1
copied 2 times b [2, 3] --------------> b 2
\
not copied c [] ------------> b 3
```
This adjustment is costly, so it's necessary to provide a way to the implementation to avoid adjusting uncaptured columns.
It's intuitive to just remove the uncaptured columns, but note in the diagram and in the query below that it can change the index of captured columns. The "c" column has index 2 in the outer scope but ends up with index 1 in the others scopes
<details><summary>Expr tree traversal with a schema with uncaptured columns removed</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ ╷
╷ a@0 = Int32 ╷
╷ b@1 = List(List(Int32)) ╷
╷ c@2 = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ b@0 = List(Int32) ╷
╷ c@1 = Int32 ╷
╷ i@2 = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ b@0 = List(Int32) ╷
╷ c@1 = Int32 ╷
╷ i@2 = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ ╷
╷ b@0 = Int32 ╷
╷ c@1 = Int32 ╷
╷ i@2 = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
```sql
select a@0, b@1, c@2, array_transform(b@0, (b@0, i@2) -> array_transform(b@0, b@0 -> b@0 + c@1 + i@2)) from t;
```
# Option 1a: Nullify uncaptured columns
To keep the indices stable, this PR won't remove uncaptured columns, as such, they are still present in the adjusted schema with their original type during tree traversals with the new _with_schema methods. However, to avoid the costly adjustment, when they are passed to the UDF in invoke_with_args, they are substituted with columns with the Null datatype.
<details><summary>Expr execution/evaluation RecordBatch schema with uncaptured columns substituted with Null columns</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ ╷
╷ a = Int32 ╷
╷ b = List(List(Int32)) ╷
╷ c = Int32 ╷
╷ ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ a = Null ! nullified ! ╷
╷ b = List(Int32) ╷
╷ c = Int32 ╷
╷ i = Int32 ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ a = Null ! nullified ! ╷
╷ b = List(Int32) ╷
╷ c = Int32 ╷
╷ i = Int32 ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ ╷
╷ a = Null ! nullified ! ╷
╷ b = Int32 ╷
╷ c = Int32 ╷
╷ i = Int32 ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
# Option 1b TreeNode *_with_indices_mapping
To avoid keeping uncaptured columns in the schema and substituting them with null in the batch, is possible to add another set of TreeNode-like methods on physical expressions that calls the visiting/transforming function with a second parameter of type HashMap<usize, usize> mapping the indices of the current scope with the ones from the outermost scope. This requires that before calling the visiting/transforming function for a physical lambda expression, all its subtree be visited to collect all the captured columns to build the indices mapping. Inner lambdas require the process again and can't reuse the work of the outer lambda. This may be costly for lambdas with complex expressions and/or highly nested.
```rust
impl PhysicalExprExt for Arc<dyn PhysicalExpr> {
pub fn transform_with_indices_mapping<
F: FnMut(Self, &HashMap<usize, usize>) -> Result<Transformed<Self>>,
>(
self,
mut f: F,
) -> Result<Transformed<Self>> {}
}
```
<details><summary>Expr tree traversal with indices_mapping: "c" has index 2 in the root scope but 1 in the others</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ ╷
╷ indices_mapping = {} ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ indices_mapping = { 1 => 2 } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ ╷
╷ indices_mapping = { 1 => 2 } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ ╷
╷ indices_mapping = { 1 => 2 } ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
The code on minimize_join_filter would change to:
```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
let mut used_columns = HashSet::new();
expr.apply_with_indices_mapping(|expr, indices_mapping| {
if let Some(col) = expr.as_any().downcast_ref::<Column>() {
// this column may be child of a lambda, where this indice would refer to the lambda
// scoped schema, which won't include uncaptured columns from the plan input,
// and therefore may differ from the indices of the schema of the input plan
// In such cases, indices_mapping contain the mapping to the index of the input plan
// if a mapping is not found, it should be a column referring to a lambda parameter
let scoped_index = col.index();
if let Some(plan_index) = indices_mapping.get(scoped_index) {
used_columns.insert(plan_index);
}
}
Ok(TreeNodeRecursion::Continue)
})
...
}
```
## Option 2. Create a schema with all parameters from all lambdas for tree traversals
Use a secondary schema containing all parameters from all lambdas. For that, expressions must be transformed, normalizing all lambda parameters and its references, with a unique qualifier per lambda, so they can coexist without conflicts in this schema. A qualifier field would be added to the lambda expr
```rust
pub struct Lambda {
pub qualifier: Option<String>,
pub params: Vec<String>,
pub body: Box<Expr>,
}
```
Schema of the example:
```
t.a: Int32
t.b: List(List(Int32))
lambda1.b: List(Int32)
lambda1.i: UInt32
lambda2.b: Int32
```
From my understanding of this [video](https://www.youtube.com/watch?v=tcFsgFXV0sM&t=1076s), this is similar to what DuckDB does on its binder, although with differences in the evaluation part. I didn't find any other resource for other open source engines with lambda support, like Clickhouse and Spark.
This works well when dealing with plans nodes, where, during plan creation time or schema recomputation, we can normalize its lambdas, create the extended schema and save it as plan field, exposing it with a method like "lambda_extended_schema", although with an added cost to plan creation/schema recomputation. The lambda normalization actually requires two passes, a first to collect any existing lambda qualifier to avoid reusing them in the last, normalizing pass.
How code would look like:
```rust
//from
expr.transform_with_schema(plan.schema(), |node, adjusted_schema| ...)
//to
let schema = plan.lambda_extended_schema();
expr.transform(|node| ...)
```
Another example:
```rust
impl LogicalPlan {
pub fn replace_params_with_values(
self,
param_values: &ParamValues,
) -> Result<LogicalPlan> {
self.transform_up_with_subqueries(|plan| {
// use plan.lambda_extended_schema() containing lambdas parameters
// instead of plan.schema() which wont
let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
let name_preserver = NamePreserver::new(&plan);
plan.map_expressions(|e| {
// if this expression is child of lambda and contain columns referring it's parameters
// the lambda_extended_schema already contain them
let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
....
```
However, when working with functions/methods that deal directly with expressions, detached from a plan, the expression lambdas may be unnormalized, and the extended schema is unavailable. There's a few public methods/functions like that, like infer_placeholder_types for example:
```rust
impl Expr {
pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
let mut has_placeholder = false;
self.transform(|mut expr| ...)
...
}
}
```
It could either:
1: Require to be only called with normalized expressions, and that the schema argument be the extended schema, or return an error otherwise, which is restrictive and put strain on users
2: Allow to be called with unnormalized expressions, visit the whole expr tree collecting the existing lambdas qualifiers to avoid to avoid duplicate qualifiers in the next step, perform a first transformation to guarantee that the expression lambdas are normalized, create the extended schema, for only then perform the second transformation to infer the placeholder types using the extended schema. While it can document that the returned expression is normalized, it's still a regular Expr which doesn't encode that property in its type. Also, without changing the method signature, it wouldn't even be possible to return the extended schema to allow it to be used again in other places without recomputation. This is costly and won't allow reuse of its costly work
<br>
<br>
Normalized example:
```sql
select t.a, t.b, array_transform(t.b, (lambda1.b, lambda1.i) -> array_transform(lambda1.b, lambda2.b -> lambda2.b + t.a + lambda1.i)) from t;
```
Just like with the first option, this also sets uncaptured columns to Null, as well as unavailable/out-of-scope lambdas parameters.
<details><summary>Expr tree batch evaluation with a single extended schema</summary>
```sql
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, (b, i) -> array_transform(b, b -> b + c + i)) ╷
╷ ╷
╷ t.a = Int32 ╷
╷ t.b = List(List(Int32)) ╷
╷ t.c = Int32 ╷
╷ lambda1.b = Null ╷
╷ lambda1.i = Null ╷
╷ lambda2.b = Null ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ (b, i) -> array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ t.a = Null ╷
╷ t.b = Null ╷
╷ t.c = Int32 ╷
╷ lambda1.b = List(Int32) ╷
╷ lambda1.i = Int32 ╷
╷ lambda2.b = Null ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ array_transform(b, b -> b + c + i) ╷
╷ ╷
╷ t.a = Null ╷
╷ t.b = Null ╷
╷ t.c = Int32 ╷
╷ lambda1.b = List(Int32) ╷
╷ lambda1.i = Int32 ╷
╷ lambda2.b = Null ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
│
▼
┌╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┐
╷ b -> b + c + i ╷
╷ ╷
╷ t.a = Null ╷
╷ t.b = Null ╷
╷ t.c = Int32 ╷
╷ lambda1.b = Null ╷
╷ lambda1.i = Int32 ╷
╷ lambda2.b = Int32 ╷
└╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶╶┘
```
</details>
With this option, indices are always stable across the whole tree
This option allows checking if a physical column refers to a lambda parameter by checking if its index is greater or equal than the number of fields of the outer schema. When this information is available, it eliminates the need to use the with_lambdas_params variations of TreeNode methods.
### Comparison between options:
| * | per-lambda schema with uncaptured columns set to null | per-lambda schema with indices_mapping | single extended schema |
|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------:|------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| New set of TreeNode methods | Yes, 1, _with_schema for both logical and physical expressions | Yes, 2, _with_schema for both logical and physical expressions, and _with_indices_mapping for physical expressions | No |
| Tree traversal added cost | Only when encountering a lambda | Only when encountering a lambda | Zero |
| Plan creation/ recompute schema added cost | Zero | Zero | Always, regardless of existence of any lambda |
| Code change, internal | New set of TreeNode methods and using them instead of the current ones when applicable | 2 new set of TreeNode methods and using them instead of the current ones when applicable | Untried, unpredictable |
| Code change, downstream | if lambda support is desired, use the new TreeNode methods instead of the current ones when applicable otherwise none | if lambda support is desired, use the new TreeNode methods instead of the current ones when applicable otherwise none | Variable, medium when closely associated with a Plan, just call plan. lambda_extended_schema() Unpredictable when plan is unavailable or doesn't exist |
| Change uncaptured columns DataType to Null | Yes | No | Yes |
| Presence of unneeded Null columns in the schema during planning and optimizing and in the RecordBatch during execution as a padding/filler to keep indices stable | Yes | No | Yes |
| Stable column indices across the whole expr tree | Yes | No | Yes |
| Make _with_lambdas_params unnecessary for physical expressions if Expr::Column is used | No | Yes | No |
# Splitting this into smaller PRs
If this PR is decided to move forward, it will likely be with smaller PRs. In that case, I already planned a division shown below. It's necessary to analyze the graph, as it doesn't help with the discussion of this text, and it's included here just to show a reasonable granularity of smaller PRs I could find in case it helps decide whether to move this forward or not.
Each rectangular node is a PR. Asymmetrical nodes are a collection of smaller PRs which share the same dependencies and aren't a dependency of any other PR. Green ones can be opened immediately. Gray ones contain unmet dependencies. Merged PRs will be colored with blue.
Left-to-Right [full-screen link](https://mermaid.live/view#pako:eNqNWG2P0zgQ_itWpJO408L1fSEfkLph0YFSLkfLfbh0ZXkTtw04drAddgviv984TtK8tWwkaDKZV8_MM5P94UQipo7r7CXJDsj_uOUIrogRpd7QHbpnOUW7hDHX3HVe7iWl3L4tbrfcMqj83qrbOut__K2DQvhBL2f-396d5TAX0ILiRUCkomgx773-VLz-xLOCYfzqxEB5bG_s_74f-mKfRISh28dMuq5P0vuYoPFoZGRct_LPMqPnz18jb30bwj80H53UBmHACEes1MWsloRrgbLDURVECgbQpOHspzdvx-HWuX3U4BVaAw-RQAvfpRm7Q9PCha1zYp6ENU9lQUUHmhJ0oCyjUlVuW4nlKvwg4JRzHulEcPCASEmOWEvC1U7IFE2umyGs3_qlgbelCApK383ZVCbJA5EUvLOSpaw9GaPiFNo50nJVif32m_cOUukJlqfcdROFrQ0MeSMp1VSixcnBXonYjP2BHxJ9wPYkts6pDvzNh3CABS2DdyiGIpwPpLiUuwlT8oWiIenWKTTPrxQtQoTf23VhHK0TyGaySyCWVnZK9uXrpsvmytWw4YRDBSVKJ3yPTOu1pTbHjCJPUBmZvI3HRe20ODhknLDkO8WRYGFDs8KExxjCSvZ5oo8YiNEXqL9xT8WaKgXq15po6rqRpPCLq_rGRX3PekI2loTvqMQZIxE9CBbDvTYe920ss4wdqwL8SB9koqkp7B6jyHRqosmk-EwLbuW60gpYXyb9AL6yjfhImetqeTSSEUSE91LkGb4_4pxzqvSQMfWVIUmVYN-K44NyVWg-ECqBZFNkQgM2mwwc01R0ddZt3cGjwA-rnkMlFBVVNB697BdrryGa7XqhMQJojAusdYNYrOo3SHCubC8phfIN6kytSJaZOp72U9TU0eidST8nOX-AuHFElMYJh_N
OATYSZcq_z7xURx4ZsxQ82yVQ8JJCRVIemeo6m0kvV1qkHphQTceWMcl00dHDWS1PqYSRS8fSgpPxYnR2UtVpjsB0Lhvt761Dz9KAKcuE1DU610Dj_QWDxgf0qMTrkVGYfDZ7HM9_b7m_KkRWVO5pLWMchyGyz1PKtSpVoIXB6GeTx-m0rWG58jZhd-REHU8n3YhPw6gctiWhRNbquZzFZktoPgenYXkC4j7tpkMKalIDwZc9yk3Llt8TMvpaLPXMO7052WyPzvZTrbqKe5CyrAZ9OVe9jSV46w4B8t-hrFqUblx-MDHEc7PWFq2yY1p1Zq4fNLQMzNG2cKsB7jqKwkuCw2Pc2G6UoLl6UGWGAwYoYNCU94xiKEwD6sOIbhwgcVyivkENFcJDDg1w9yfhx-YLNO5rKIxpUU-NAWyyr8wsK8bWgJICsEoNJ-CKzcyyIqNLIoCOMX2kg5Lzy3sC-tUigX-1SAyFI2mxCJwzUKMFhrE8lPrBcBkAchGTwvQbYTkxgwZCVjnTGO5omoFP90RHhyEFDZ-UOR0OC-iZtSMxeweURZarA47FA8fWOqxEgmsCk24gZrtZr_P7rzmVx414L4Bv1te_MSW5ZAlR5fIzOPuq-bQ-SBrHNH6v6mVJWvQ_t2t0caZs9adsEufbPqjbPmi0_ROUdNu_ozJ8ioqnAMGNuXpAYKf70GCf9vcSIiFzOoCcm5R7prbNBtDnNGghdjtFten7osv6THkWk2pPHaiycqnFQsKqbDalgRoozoLTh2qVGEYWZtRcAB9Y4mGp3_PClUuM8N1OoJWGTmt2NgDzMZKVR2YT1NmrzQ5QM1zWMySuwf0Uf4ZewrlOAMIHVeTcHGEGrQKea4qqr4QSHs8AYdVjZeD_AqbQoegnl9rNuXL2Mokd-ODI6ZWTUsBO8-j8MExbR0Pq6NZx4TYm8oup0J8gkxH-nxBpJQYfKPuD4-4IU_Bka-dNQqBbTyxgjkrPDCbHnRca
HPeH8-i409H1i9F8Mno1nc4ms8liPL1yjsAzfTFfTOaj8XQ8G79azK9_XjnfC5ujFy-vQQOclhZyZf_KU_yx5-f_S_hr8A)
<details>
```mermaid
graph LR
classDef blue fill:blue
classDef green fill:green
subgraph "SQL" [SQL 84LOC]
SQLP[SQL Parse 65LOC]
SQLU[SQL Unparse 19LOC]
end
LL[Logical Expr::Lambda 100LOC]:::green
LL --> CSE[CSE 50LOC]
P[Plan logical lambda into physical expr 25LOC]
UDF1["Extend ScalarUDF[Impl] 300LOC"]
UDF2[ScalarUDF lambda schema helpers 100LOC]
AM[Non functional array_transform 270LOC]
PSFL[ScalarFunction PhysicalExpr lambda aware 30LOC]
PL --> PSFL
UDF1 --> PSFL
UDF1 --> AM
%%CILP[Column::is_lambda_parameter 6LOC]
subgraph "Expr::*_with_schema"
LTN[Expr::*_with_schema API def 50LOC]:::green
LTNB[make Expr::*_with_schema lambda aware 70LOC]
LTN --> LTNES[Expr Simplifier 100LOC]
LTNA>"
use Expr::*_with_schema in existing code
Type Coercion 110LOC
normalize_col[with_schemas_and_ambiguity_check] 31LOC
SessionState::create_physical_expr 4LOC
Expr::infer_placeholder_type 1LOC
ApplyFunctionRewrites 10LOC
optmize_projections::rewrite_expr 2LOC
SqlToRel::try_process_group_by_unnest 10LOC
sql resolve_columns 5LOC
Example type_coercion_demo 10LOC
"]
end
PL[Physical Lambda Expr 108LOC]:::green
subgraph "PhysicalExpr::*_with_schema"
PTN[PhysicalExpr::*_with_schema API def 25LOC]:::green
PTNA>"
use PhysicalExpr::*_with_schema in ProjectionMapping 32LOC
PhysicalExprSimplifier 20LOC
unwrap_cast_in_comparison 10LOC
AsyncMapper::find_references 1LOC
Example CustomCastsPhysicalExprAdapter 10LOC
"]
PTNB[make PhysicalExpr::*_with_schema lambda aware 160LOC]
end
subgraph capture
CS[Capture support 30LOC]
LCH["List capture helpers 60LOC(4x15)"]
MCH["Merge capture with arguments helper 66LOC(2x33)"]
AMCT[array_transform capture support 20LOC]
end
PSFL --> CS
PTN --> CS
LL --> SQL
LL --> P
UDF2 --> LTNES
UDF2 --> LTNB
UDF2 --> PTNB
LTN --> LTNA
LTN --> LTNB
LL --> LTNB
LTN --> UDF2
LL --> UDF1 --> UDF2
UDF2 --> P
PL --> P
PL --> PTNB
PTN --> PTNB
PTN --> PTNA
AM --> AMCT
CS --> AMCT
LCH --> AMCT
MCH --> AMCT
LL --> LTNLP2
subgraph "Expr::*_with_lambdas_params"
LTNLP --> LTNLP2[make Expr::*_with_lambdas_params lambda aware]
LTNLP[Expr::*_with_lambdas_params API def 50LOC]:::green -->
AAAAA>"
expr_applicable_for_cols 5LOC
Expr::add_column_refs[_counts]/any_column_refs 15LOC
expr_to_columns 10LOC
columnize_expr 15LOC
find_columns_referenced_by_expr 10LOC
find_column_indexes_referenced_by_expr 5LOC
normalize_col 10LOC
normalize_col_with_schemas_and_ambiguity_check 15LOC
replace_col 10LOC
transform_up_with_lambdas_params 10LOC
filter_exprs_evaluation_result_on_empty_batch 10LOC
replace_cols_by_name 10LOC
optimizer::push_down_filter::contain 15LOC
ScalarSubqueryToJoin 40LOC
TableAliasRewriter 20LOC
Example ShreddedJsonRewriter 30LOC
"]
end
PL --> PTNLP2
subgraph "PhysicalExpr::*_with_lambdas_params"
PTNLP --> PTNLP2[make PhysicalExpr::*_with_lambdas_params lambda aware]
PTNLP[PhysicalExpr::*_with_lambdas_params API def 50LOC]:::green -->
BBBBB>"
CustomPhysicalExprAdapter 3LOC
ParquetPushdownChecker 13LOC
add_offset_to_expr 3LOC
update_expr 10LOC
project_ordering 20LOC
with_new_schema 10LOC
collect_columns 10LOC
reassign_expr_columns 10LOC
DefaultPhysicalExprAdapter 40LOC
projection pushdown 50LOC
sort pushdown 40LOC
projection 50LOC
stream_join_utils 40LOC
pruning predicate rewrite_column_expr 5LOC
Example DefaultValuePhysicalExprAdapter 20LOC
"]
end
```
</details>
<br>
<br>
Top-to-Bottom [full-screen link](https://mermaid.live/view#pako:eNqNWG2P0zgQ_itWpJO408K12baUfEDqhkUHarkcLffh0pXlTdw24NjBdtgtiP9-4zhJ89aykaDJZF49M89M9ocTiZg6nrOXJDugzc2WI7giRpR6Q3fonuUU7RLGPHPXebmXlHL7trjdcsug8nurbuus_1luHRTCD5pPln_7d5bDXEALihcBkYqi2bT3-lPx-hPPCobxqxMD5bG9sf8vl-FS7JOIMHT7mEnPW5L0PiZoPBoZGc-r_LPM6Pnz18hf34bwD01HJ7VBGDDCESt1Masl4Vqg7HBUBZGCAeQ2nP305u043Dq3jxq8QmvgIRJo4bs0Y3founBh65yY3bDmqSyo6EBTgg6UZVSqym0rsViFHwSccs4jnQgOHhApyRFrSbjaCZki92UzhPXbZWngbSmCgtJ3czaVSfJAJAXvrGQpa0_GqDiFdo60WFViv_3mv4NU-oLlKfe8RGFrA0PeSEo1lWh2crBXIjZjf-CHRB-wPYmtc6qD5eZDOMCCFsE7FEMRTgdSXMrdhCn5QtGQdOsUmudXihYhwu_tujCO1glkM9klEEsrOyX74nXTZXPlathwwqGCEqUTvkem9dpSm2NGkS-ojEzexuOidlocHDJOWPKd4kiwsKFZYcJjDGEl-zzRRwzE6AvU37inYk2VAvVrTTT1vEhS-MVVfeOivic9IRtLwndU4oyRiB4Ei-FeG4_7NhZZxo5VAX6kDzLR1BR2j1FkOjXRZFJ8pgW38jxpBawvbj-Ar2wjPlLmeVoejWQEEeG9FHmG748455wqPWRMfWVIUiXYt-L4oFwVmg6ESiDZFJnQgM0mA8c0FV2ddVt38ChYhlXPoRKKiioaj-b9Yu01RLNdLzRGAI1xgbVuEItV_QYJzpXtJaVQvkGdqRXJMlPH1_0UNXU0esft5yTnDxA3jojSOOF
w3inARqJM-feZF-rII2OWgme7BApeUqhIyiNTXWcz6edKi9QHE6rp2CImmS46ejir5SmVMHLpWFpwMp6Nzk6qOs0RmM5lo_39dehbGjBlmZC6RucaaPy_YNAsAT0q8XpkFCafTR7H099b7q8KkRWVe1rLGMdhiOzzlHKtShVoZjD6mft4fd3WsFj5m7A7cqKOp2434tMwKodtSSiRtXouZ7HZEprPwWlYnoC4T7vpkIKa1EDwRY9y07K17AkZfS2Weuad3pxstkdn-6lWXcU9SFlUg76cq_7GEvx1hwD571BWLUo3rmXgGuK5WWuLVtkxrTozdxk0tAzM0bZwqwHuOorCS4LDY9zYbpSguXpQZYYDBihg0JT3jGIoTAPqw4huHCBxXKK-QQ0VwkMODXD3J-HH5gs07msojGlRT40BbLKvzCwrxtaAkgKwSg0n4IrNzLIio0sigI4xfaSDktPLewL61SKBf7VIDIUjabEInDNQowWGsTyU-sFwGQByEZPC9BthOTGDBkJWOdMY7miagU_3REeHIQUNn5Q5HQ4L6Jm1IzF7B5RFlqsDjsUDx9Y6rESCawKTbiBmu1mv8_uvOZXHjXgvgG_S178xJblgCVHl8jM4-6r5tD5IGsc0fq_qZUla9D-3a3Rxpmz1p2wS59s-qNs-aLT9E5R027-jMnyKiqcAwY25ekBgp_vQYL_u7yVEQuZ0ADk3KfdNbZsNoM9p0ELsdopq0_dFl_WZ8iwm1Z46UGXlUouFhFXZbEoDNVCcBacP1SoxjCzMqLkAPrDEw1K_54Urlxjhu51AKw2d1uRsAOZjJCuPzCaos1ebHaBmuKxnSFyD-yn-DL2Ec50AhA-qyLk5wgxaBTzXFFVfCSU8ngHCqsfKwP8FTKFD0buX2s25cvYyiR344MjplZNSwE7z6PwwTFtHQ-ro1vHgNibyi6nQnyCTEf6fEGklBh8o-4Pj7QhT8GRr501CoFtPLGCOSt8MJseb
Fhoc74fzCA_TF6OpOxqN5pPpZDx151fO0fHG7vzFdOZOJ_PJ7Ho2mb1yf1453wuboxfzl6ABTksLubJ_5Sn-2PPzf8Ata_o)
<details>
```mermaid
graph TB
classDef blue fill:blue
classDef green fill:green
subgraph "SQL" [SQL 84LOC]
SQLP[SQL Parse 65LOC]
SQLU[SQL Unparse 19LOC]
end
LL[Logical Expr::Lambda 100LOC]:::green
LL --> CSE[CSE 50LOC]
P[Plan logical lambda into physical expr 25LOC]
UDF1["Extend ScalarUDF[Impl] 300LOC"]
UDF2[ScalarUDF lambda schema helpers 100LOC]
AM[Non functional array_transform 270LOC]
PSFL[ScalarFunction PhysicalExpr lambda aware 30LOC]
PL --> PSFL
UDF1 --> PSFL
UDF1 --> AM
%%CILP[Column::is_lambda_parameter 6LOC]
subgraph "Expr::*_with_schema"
LTN[Expr::*_with_schema API def 50LOC]:::green
LTNB[make Expr::*_with_schema lambda aware 70LOC]
LTN --> LTNES[Expr Simplifier 100LOC]
LTNA>"
use Expr::*_with_schema in existing code
Type Coercion 110LOC
normalize_col[with_schemas_and_ambiguity_check] 31LOC
SessionState::create_physical_expr 4LOC
Expr::infer_placeholder_type 1LOC
ApplyFunctionRewrites 10LOC
optmize_projections::rewrite_expr 2LOC
SqlToRel::try_process_group_by_unnest 10LOC
sql resolve_columns 5LOC
Example type_coercion_demo 10LOC
"]
end
PL[Physical Lambda Expr 108LOC]:::green
subgraph "PhysicalExpr::*_with_schema"
PTN[PhysicalExpr::*_with_schema API def 25LOC]:::green
PTNA>"
use PhysicalExpr::*_with_schema in ProjectionMapping 32LOC
PhysicalExprSimplifier 20LOC
unwrap_cast_in_comparison 10LOC
AsyncMapper::find_references 1LOC
Example CustomCastsPhysicalExprAdapter 10LOC
"]
PTNB[make PhysicalExpr::*_with_schema lambda aware 160LOC]
end
subgraph capture
CS[Capture support 30LOC]
LCH["List capture helpers 60LOC(4x15)"]
MCH["Merge capture with arguments helper 66LOC(2x33)"]
AMCT[array_transform capture support 20LOC]
end
PSFL --> CS
PTN --> CS
LL --> SQL
LL --> P
UDF2 --> LTNES
UDF2 --> LTNB
UDF2 --> PTNB
LTN --> LTNA
LTN --> LTNB
LL --> LTNB
LTN --> UDF2
LL --> UDF1 --> UDF2
UDF2 --> P
PL --> P
PL --> PTNB
PTN --> PTNB
PTN --> PTNA
AM --> AMCT
CS --> AMCT
LCH --> AMCT
MCH --> AMCT
LL --> LTNLP2
subgraph "Expr::*_with_lambdas_params"
LTNLP --> LTNLP2[make Expr::*_with_lambdas_params lambda aware]
LTNLP[Expr::*_with_lambdas_params API def 50LOC]:::green -->
AAAAA>"
expr_applicable_for_cols 5LOC
Expr::add_column_refs[_counts]/any_column_refs 15LOC
expr_to_columns 10LOC
columnize_expr 15LOC
find_columns_referenced_by_expr 10LOC
find_column_indexes_referenced_by_expr 5LOC
normalize_col 10LOC
normalize_col_with_schemas_and_ambiguity_check 15LOC
replace_col 10LOC
transform_up_with_lambdas_params 10LOC
filter_exprs_evaluation_result_on_empty_batch 10LOC
replace_cols_by_name 10LOC
optimizer::push_down_filter::contain 15LOC
ScalarSubqueryToJoin 40LOC
TableAliasRewriter 20LOC
Example ShreddedJsonRewriter 30LOC
"]
end
PL --> PTNLP2
subgraph "PhysicalExpr::*_with_lambdas_params"
PTNLP --> PTNLP2[make PhysicalExpr::*_with_lambdas_params lambda aware]
PTNLP[PhysicalExpr::*_with_lambdas_params API def 50LOC]:::green -->
BBBBB>"
CustomPhysicalExprAdapter 3LOC
ParquetPushdownChecker 13LOC
add_offset_to_expr 3LOC
update_expr 10LOC
project_ordering 20LOC
with_new_schema 10LOC
collect_columns 10LOC
reassign_expr_columns 10LOC
DefaultPhysicalExprAdapter 40LOC
projection pushdown 50LOC
sort pushdown 40LOC
projection 50LOC
stream_join_utils 40LOC
pruning predicate rewrite_column_expr 5LOC
Example DefaultValuePhysicalExprAdapter 20LOC
"]
end
```
</details>
<br>
<br>
Right-to-Left [full-screen link](https://mermaid.live/view#pako:eNqNWG2PmzgQ_isWUqXeadsL5G3Lh0pZutW1SlquSe_DkZXlBSehBUNt09206n-_sQ2Et6SL1AaGefXMPDPsTyvMImq51p6T_IA-LbcMwRUmRIg3dIfuk4KiXZwkrrrrvNxzSpl5q2-3zDCI4t6o21rrf5ZbCwXwg64ny4_eneFQF9B8_cInXFA0m_Zef9avP7NcM9ivTgyURebG_L9cBstsH4ckQbePOXfdJUnvI4Ls0UjJuG7ln2FGL168Rt76NoB_aDo6qfUDPyEMJaWuxGiJmcxQfjgKTaRgADkNZz-_eWsHW-v2UYJXaA08hAMteJfmyR0aaxe21onZCWqeyoIIDzQl6ECTnHJRuW0kFqvgQwanXLBQxhkDDwjn5IglJ0zsMp4iZ94MYf12WRp4W4ogv_RdnU1lkjwQTsE7I1nKmpNRKk6hnSMtVpXYs2feO0illyVFylw3FtjYwJA3klJJOZqdHOyViMnYn_ghlgdsTmJrnepgufkQDLCghf8ORVCE04EUl3I3QUq-UjQk3TqF5vmVojpE-L1da-NoHUM2410MsbSyU7IvXjddVlchhg3HDCooFjJme6Rary21OeYUeRnlocqbbevaaXEwyDhJ4h8Uh1kSNDQLTFiEIax4X8TyiIEYfoX6s3sq1lQIUL-WRFLXDTmFX1zVN9b1PekJmVhitqMc5wkJ6SFLIriXyuO-jUWeJ8eqAD_RBx5Lqgq7x5jlMlXR5Dz7QjW3cF1uBIwvTj-Ab8km-0QT15X8qCRDiAjveVbk-P6IC8aokEPGxLcEcSqy5Ls-PihXgaYDoRJINkUqNGAzycARTbOuzrqtO3jkL4Oq51AJRbqK7NF1v1h7DdFs1wuN4UNjXGCtG8RgVb9B_HNle0kplK9fZ2pF8lzV8bifoqaORu84_ZwU7AHixiEREsc
MzjsF2IiFKv8-80IcWajMUvBsF0PBcwoVSVmoqutsJr1CyCz1wIRoOraISC51Rw9ntTylEkYuHUsLTuzZ6OykqtMcgumCN9rfWweeoQFTnmdc1uhcA433NwyaJaBHJV6PDG3y-eTRnv7Rcn-lRVaU72ktoxyHIbIvUsqkKFWgmcLo587jeNzWsFh5m6A7csKOp0434tMwKodtSSiRtXouZ7HaEprP_mlYnoC4T7vpkPya1EDwRY9y07K17AkpfS2Weuad3pxstkdn-6lWXcU9SFlUg76cq97GELx1hwD571BWLUo3rqXvKOK5WWuKVpgxLTozd-k3tAzM0bZwqwHuOoqCS4LDY1zZbpSgunpQpYYDBihIoCnvE4qhMBWoDyO6coBEUYn6CjVEAA8FNMDdX4Qdmy-Q3degjcmsnhoD2GReqVmmx9aAEg1YpYYTcEVqZhmR0SURQMeIPtJByenlPQH9bpHAv1skhsLhVC8C5wzUaIFhLA-lfjDcBABZxyQw_U6SgqhBAyGLIpEY7miag0_3RIaHIQUNn4Q6HQYL6Jm1I1Z7B5RFXogDjrIHho11WIkyJglMuoGYzWa9Lu6_FZQfN9n7DPgmff0bVZKLJCaiXH4GZ181n9YHTqOIRu9FvSxxg_7ndo0uzpSt_pRN4nzb-3Xb-422f4KSbvt3VAZPUfEUILhRVw8IzHQfGuzj_l5COGRO-pBzlXJP1bbaAPqcCi2y3U5Qqfped1mfqcgjUu2pA1VWLrU447Aqq01poAb0WTD6UK0Sw8iSKDUXwAeWeFjq90y7cokRvtsJtNLQaU3OBqA-RvLyyEyCOnu12gFqhst6hsQluJ_iL9BLuJAxQPigioKpI8yhVcBzSVH1lVDC4xkgrHqsDPxfwBQ6FL1zqd2sK2vP48iCD46CXlkpBexUj9ZPxbS1JKSObi0XbiPCv6oK_QUyOWH_ZVlaicEHyv5guTuSCHgytfMmJtCtJxYwR7mnBpPl
TsZaheX-tB4t15lPX85H9tieje3pK2fmTK-so-W-cGavXtrX89FcE8fXzuzXlfVDm7VfTp35xJmPJ_PpfDSb2fMrC05PZnxl_uqj__jz63_6m29E)
<details>
```mermaid
graph RL
classDef blue fill:blue
classDef green fill:green
subgraph "SQL" [SQL 84LOC]
SQLP[SQL Parse 65LOC]
SQLU[SQL Unparse 19LOC]
end
LL[Logical Expr::Lambda 100LOC]:::green
LL --> CSE[CSE 50LOC]
P[Plan logical lambda into physical expr 25LOC]
UDF1["Extend ScalarUDF[Impl] 300LOC"]
UDF2[ScalarUDF lambda schema helpers 100LOC]
AM[Non functional array_transform 270LOC]
PSFL[ScalarFunction PhysicalExpr lambda aware 30LOC]
PL --> PSFL
UDF1 --> PSFL
UDF1 --> AM
%%CILP[Column::is_lambda_parameter 6LOC]
subgraph "Expr::*_with_schema"
LTN[Expr::*_with_schema API def 50LOC]:::green
LTNB[make Expr::*_with_schema lambda aware 70LOC]
LTN --> LTNES[Expr Simplifier 100LOC]
LTNA>"
use Expr::*_with_schema in existing code
Type Coercion 110LOC
normalize_col[with_schemas_and_ambiguity_check] 31LOC
SessionState::create_physical_expr 4LOC
Expr::infer_placeholder_type 1LOC
ApplyFunctionRewrites 10LOC
optmize_projections::rewrite_expr 2LOC
SqlToRel::try_process_group_by_unnest 10LOC
sql resolve_columns 5LOC
Example type_coercion_demo 10LOC
"]
end
PL[Physical Lambda Expr 108LOC]:::green
subgraph "PhysicalExpr::*_with_schema"
PTN[PhysicalExpr::*_with_schema API def 25LOC]:::green
PTNA>"
use PhysicalExpr::*_with_schema in ProjectionMapping 32LOC
PhysicalExprSimplifier 20LOC
unwrap_cast_in_comparison 10LOC
AsyncMapper::find_references 1LOC
Example CustomCastsPhysicalExprAdapter 10LOC
"]
PTNB[make PhysicalExpr::*_with_schema lambda aware 160LOC]
end
subgraph capture
CS[Capture support 30LOC]
LCH["List capture helpers 60LOC(4x15)"]
MCH["Merge capture with arguments helper 66LOC(2x33)"]
AMCT[array_transform capture support 20LOC]
end
PSFL --> CS
PTN --> CS
LL --> SQL
LL --> P
UDF2 --> LTNES
UDF2 --> LTNB
UDF2 --> PTNB
LTN --> LTNA
LTN --> LTNB
LL --> LTNB
LTN --> UDF2
LL --> UDF1 --> UDF2
UDF2 --> P
PL --> P
PL --> PTNB
PTN --> PTNB
PTN --> PTNA
AM --> AMCT
CS --> AMCT
LCH --> AMCT
MCH --> AMCT
LL --> LTNLP2
subgraph "Expr::*_with_lambdas_params"
LTNLP --> LTNLP2[make Expr::*_with_lambdas_params lambda aware]
LTNLP[Expr::*_with_lambdas_params API def 50LOC]:::green -->
AAAAA>"
expr_applicable_for_cols 5LOC
Expr::add_column_refs[_counts]/any_column_refs 15LOC
expr_to_columns 10LOC
columnize_expr 15LOC
find_columns_referenced_by_expr 10LOC
find_column_indexes_referenced_by_expr 5LOC
normalize_col 10LOC
normalize_col_with_schemas_and_ambiguity_check 15LOC
replace_col 10LOC
transform_up_with_lambdas_params 10LOC
filter_exprs_evaluation_result_on_empty_batch 10LOC
replace_cols_by_name 10LOC
optimizer::push_down_filter::contain 15LOC
ScalarSubqueryToJoin 40LOC
TableAliasRewriter 20LOC
Example ShreddedJsonRewriter 30LOC
"]
end
PL --> PTNLP2
subgraph "PhysicalExpr::*_with_lambdas_params"
PTNLP --> PTNLP2[make PhysicalExpr::*_with_lambdas_params lambda aware]
PTNLP[PhysicalExpr::*_with_lambdas_params API def 50LOC]:::green -->
BBBBB>"
CustomPhysicalExprAdapter 3LOC
ParquetPushdownChecker 13LOC
add_offset_to_expr 3LOC
update_expr 10LOC
project_ordering 20LOC
with_new_schema 10LOC
collect_columns 10LOC
reassign_expr_columns 10LOC
DefaultPhysicalExprAdapter 40LOC
projection pushdown 50LOC
sort pushdown 40LOC
projection 50LOC
stream_join_utils 40LOC
pruning predicate rewrite_column_expr 5LOC
Example DefaultValuePhysicalExprAdapter 20LOC
"]
end
```
</details>
<br>
<br>
Bottom-to-Top [full-screen link](https://mermaid.live/view#pako:eNqNWFuP0zgU_itWpJXY1cC2naaUPCB1wqAFtWyWln3YdGR5ErcNJHawHWYK4r_vcZykubhlIkET59wv3zmZH07EY-p4zl6Q_IBuNluG4IpSIuUbukP3aUHRLklTT9_1Xu4Fpcy8LW-3zBDI4t6I2zrrf5ZbB4Xwg-bT5d_-naHQF5wF5YuACEnRzB28_lS-_sTykmD86kRAWWxuzP_LZbjk-yQiKbp9zIXnLUl2HxM0Ho00j-fV9hli9Pz5a-Svb0P4h9zRSWwQBilhKK1kpUZKwhRH-eEoy0MKCtCkZeynN2_H4da5fVRgFVoDDRFwFr7L8vQOXZcmbJ0T8SRsaGoNMjrQjKADTXMqZG224Viswg8colywSCWcgQVECHLEShAmd1xkaPKy7cL67bJS8LZiQUFlu45NrZI8EEHBOsNZ8ZrIaBEn184dLVY122-_-e8glT5Pi4x5XiKx0YEhbySjigo0Oxk4KBGTsT_wQ6IO2ERi65zqYLn5EFpI0CJ4h2IoQteS4orvJszIF4ps3J0otONXsZYuwu_tulSO1glkM9kl4EsnOxX54nXbZH0V0q44YVBBiVQJ2yPdel2uzTGnyOdURDpv43FZOx0KBhknafKd4oinYUuyxITFGNxK9kWijhgOoy9Qf-OBiDWVEsSvFVHU8yJB4RfX9Y3L-p4OmIwvCdtRgfOURPTA0xjulbZ4qGOR5-mxLsCP9EEkiurCHhDyXGXam1zwz7Sklp4nDIOxZTJ04Gu64R9p6nlKHDVnBB7hveBFju-PuGCMSmVTJr-mSFDJ029l-KBcJXItrhJINkXaNSAzycAxzXhfZtPWPTwKlmHdc6iCorKKxqP5sFgHDdFu1wuNEUBjXCBtGsRg1bBBgnNle0kolG_QZGpF8lzX8fUwRW0Zrd6ZDHNSsAfwG0dEKpw
wiHcGsJFIXf5D4oU8skirpWDZLoGCFxQqkrJIV9fZTPqFVDzzQYVsG7aISa7KjrZntYpSBSOXwtKBk_FsdHZSNWmOQHUhWu3vr0PfnAFRnnOhGnRugMb_CwbNEtCjZm9GRqny2fRx7P7eMX9Vsqyo2NOGRxsOQ2RfZJQpWYlAM43RzyaP19ddCYuVvwn7IyfqWTrpe3waRtWwrQ4qZK2fq1mst4T2c3AalicgHp7d9I6C5qiF4IvByU1H13LApOV1SJqZd3pz0tkdnd2nRnTtt_VkUQ_6aq761RLmr3sHkP_eyapz0vdrGUz04blZa4pWmjEtezN3GbSkWOZol7nTAHc9QeElRvsY17pbJaivAVTp4YABClJoyvuUYihMDep2RNcGkDiuUF-jhgzhoYAGuPuTsGP7BRoPJZTKFG-mhgWbzCs9y8qxZRFSAlYl4QRcsZ5ZhmV0iQXQMaaP1MrpXt4T0K8WCfyrRcLmjqDlInBOQYMWGMayLfVWd1MA5NIniek3khZEDxpwWRapwnBHsxxsuicqOtgEtGySOjoMFtAza0ei9w4oi7yQBxzzB4aNdliJOFMEJp3FZ7NZr4v7rwUVxw1_z4FuOpS_0SW5SBMiq-XHOvvq-bQ-CBrHNH4vm2VJGPQ_t2v0caZq9adsEufbPmjaPmi1_ROE9Nu_JzJ8ioinAMGNvgZAYKa7bbBfD_cSIiBzKoCc65T7urb1BjCk1GjBdztJle77ssuGREUek3pPtVRZtdRiLmBV1puSpQbKWDD6UK8SdmRJtZgL4ANLPCz1e1aacokQvtsJtJItWtOzDuiPkbwKmUlQb6_WO0BDcFmOjV2B-Rn-DL2EC5UAhFtFFEyHMIdWAcsVRfVXQgWPZ4Cw7rHK8X8BU6jN-8mldnOunL1IYgc-OAp65WQUsFM_Oj800dZRkDq6dTy4jYn4oiv0J_DkhP3HeVazwQfK_uB4O5JKeDK18yYh0K0nElBHha8Hk-O5
o1Epw_F-OI_w6L4YuZPRaDSfutOxO5lfOUfHG0_mL9zZxJ3Op7Pr2XT2avLzyvleah29mL90rxyIl-JiZf7OU_655-f_jC5sWg)
<details>
```mermaid
graph BT
classDef blue fill:blue
classDef green fill:green
subgraph "SQL" [SQL 84LOC]
SQLP[SQL Parse 65LOC]
SQLU[SQL Unparse 19LOC]
end
LL[Logical Expr::Lambda 100LOC]:::green
LL --> CSE[CSE 50LOC]
P[Plan logical lambda into physical expr 25LOC]
UDF1["Extend ScalarUDF[Impl] 300LOC"]
UDF2[ScalarUDF lambda schema helpers 100LOC]
AM[Non functional array_transform 270LOC]
PSFL[ScalarFunction PhysicalExpr lambda aware 30LOC]
PL --> PSFL
UDF1 --> PSFL
UDF1 --> AM
%%CILP[Column::is_lambda_parameter 6LOC]
subgraph "Expr::*_with_schema"
LTN[Expr::*_with_schema API def 50LOC]:::green
LTNB[make Expr::*_with_schema lambda aware 70LOC]
LTN --> LTNES[Expr Simplifier 100LOC]
LTNA>"
use Expr::*_with_schema in existing code
Type Coercion 110LOC
normalize_col[with_schemas_and_ambiguity_check] 31LOC
SessionState::create_physical_expr 4LOC
Expr::infer_placeholder_type 1LOC
ApplyFunctionRewrites 10LOC
optmize_projections::rewrite_expr 2LOC
SqlToRel::try_process_group_by_unnest 10LOC
sql resolve_columns 5LOC
Example type_coercion_demo 10LOC
"]
end
PL[Physical Lambda Expr 108LOC]:::green
subgraph "PhysicalExpr::*_with_schema"
PTN[PhysicalExpr::*_with_schema API def 25LOC]:::green
PTNA>"
use PhysicalExpr::*_with_schema in ProjectionMapping 32LOC
PhysicalExprSimplifier 20LOC
unwrap_cast_in_comparison 10LOC
AsyncMapper::find_references 1LOC
Example CustomCastsPhysicalExprAdapter 10LOC
"]
PTNB[make PhysicalExpr::*_with_schema lambda aware 160LOC]
end
subgraph capture
CS[Capture support 30LOC]
LCH["List capture helpers 60LOC(4x15)"]
MCH["Merge capture with arguments helper 66LOC(2x33)"]
AMCT[array_transform capture support 20LOC]
end
PSFL --> CS
PTN --> CS
LL --> SQL
LL --> P
UDF2 --> LTNES
UDF2 --> LTNB
UDF2 --> PTNB
LTN --> LTNA
LTN --> LTNB
LL --> LTNB
LTN --> UDF2
LL --> UDF1 --> UDF2
UDF2 --> P
PL --> P
PL --> PTNB
PTN --> PTNB
PTN --> PTNA
AM --> AMCT
CS --> AMCT
LCH --> AMCT
MCH --> AMCT
LL --> LTNLP2
subgraph "Expr::*_with_lambdas_params"
LTNLP --> LTNLP2[make Expr::*_with_lambdas_params lambda aware]
LTNLP[Expr::*_with_lambdas_params API def 50LOC]:::green -->
AAAAA>"
expr_applicable_for_cols 5LOC
Expr::add_column_refs[_counts]/any_column_refs 15LOC
expr_to_columns 10LOC
columnize_expr 15LOC
find_columns_referenced_by_expr 10LOC
find_column_indexes_referenced_by_expr 5LOC
normalize_col 10LOC
normalize_col_with_schemas_and_ambiguity_check 15LOC
replace_col 10LOC
transform_up_with_lambdas_params 10LOC
filter_exprs_evaluation_result_on_empty_batch 10LOC
replace_cols_by_name 10LOC
optimizer::push_down_filter::contain 15LOC
ScalarSubqueryToJoin 40LOC
TableAliasRewriter 20LOC
Example ShreddedJsonRewriter 30LOC
"]
end
PL --> PTNLP2
subgraph "PhysicalExpr::*_with_lambdas_params"
PTNLP --> PTNLP2[make PhysicalExpr::*_with_lambdas_params lambda aware]
PTNLP[PhysicalExpr::*_with_lambdas_params API def 50LOC]:::green -->
BBBBB>"
CustomPhysicalExprAdapter 3LOC
ParquetPushdownChecker 13LOC
add_offset_to_expr 3LOC
update_expr 10LOC
project_ordering 20LOC
with_new_schema 10LOC
collect_columns 10LOC
reassign_expr_columns 10LOC
DefaultPhysicalExprAdapter 40LOC
projection pushdown 50LOC
sort pushdown 40LOC
projection 50LOC
stream_join_utils 40LOC
pruning predicate rewrite_column_expr 5LOC
Example DefaultValuePhysicalExprAdapter 20LOC
"]
end
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3651815428
Thanks @fbx31. This is high priority for me right now. I also hope we can finish this quickly. It's indeed a lot work, but now I believe it ended up that way because I went to fast in the wrong direction... More on the comment below. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2982634042
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Done at https://github.com/apache/datafusion/pull/18921/changes/a9d0e6cbf4b7ddd0e3d70a4076193f93073af62a (outdated diff https://github.com/apache/datafusion/pull/18921/changes/6ae73cb4a569d6693d37b601cca38a4c689722df), thanks for the tests
> 1. replace null lists that is not empty underneath, with nulls with empty list underneath.
...
I'm conflicted on whether we should fix case 1 for them as it is costly and the user might have prior knowledge to avoid that
I was waiting your input whether this should be handled or not. I share your concern regarding the cost of this, but also agree that it should be fixed by default, as done now. I think it can be made configurable, either via a config like `SET array_transform.remove_nulls = false` or an optional argument, and add an UDF that does that cleanup so an array can be explicit cleaned once and then be transformed multiples times across a plan without being re-cleaned again for each transformation, but not on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3002322975
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Maybe a middle ground solution would be introducing a trait method with a default value so implementors can indicate whether they want DataFusion to handle this or not:
```
fn clean_null_values(&self) -> bool {
true
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2966333019
##########
datafusion/expr/src/udlf.rs:
##########
@@ -0,0 +1,550 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`]: Lambda User Defined Functions
+
+use crate::expr::schema_name_from_exprs_comma_separated_without_space;
+use crate::{ColumnarValue, Documentation, Expr};
+use arrow::array::{ArrayRef, RecordBatch};
+use arrow::datatypes::{DataType, Field, FieldRef, Schema};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{Result, ScalarValue, not_impl_err};
+use datafusion_expr_common::dyn_eq::{DynEq, DynHash};
+use datafusion_expr_common::signature::Volatility;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::any::Any;
+use std::cmp::Ordering;
+use std::fmt::Debug;
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+/// The types of arguments for which a function has implementations.
+///
+/// [`LambdaTypeSignature`] **DOES NOT** define the types that a user query could call the
+/// function with. DataFusion will automatically coerce (cast) argument types to
+/// one of the supported function signatures, if possible.
+///
+/// # Overview
+/// Functions typically provide implementations for a small number of different
+/// argument [`DataType`]s, rather than all possible combinations. If a user
+/// calls a function with arguments that do not match any of the declared types,
+/// DataFusion will attempt to automatically coerce (add casts to) function
+/// arguments so they match the [`LambdaTypeSignature`]. See the [`type_coercion`] module
+/// for more details
+///
+/// [`type_coercion`]: crate::type_coercion
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub enum LambdaTypeSignature {
+ /// The acceptable signature and coercions rules are special for this
+ /// function.
+ ///
+ /// If this signature is specified,
+ /// DataFusion will call [`LambdaUDF::coerce_value_types`] to prepare argument types.
+ UserDefined,
+ /// One or more lambdas or arguments with arbitrary types
+ VariadicAny,
+ /// The specified number of lambdas or arguments with arbitrary types.
+ Any(usize),
+}
+
+/// Provides information necessary for calling a lambda function.
+///
+/// - [`LambdaTypeSignature`] defines the argument types that a function has implementations
+/// for.
+///
+/// - [`Volatility`] defines how the output of the function changes with the input.
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub struct LambdaSignature {
+ /// The data types that the function accepts. See [LambdaTypeSignature] for more information.
+ pub type_signature: LambdaTypeSignature,
+ /// The volatility of the function. See [Volatility] for more information.
+ pub volatility: Volatility,
+}
+
+impl LambdaSignature {
+ /// Creates a new `LambdaSignature` from a given type signature and volatility.
+ pub fn new(type_signature: LambdaTypeSignature, volatility: Volatility) -> Self {
+ LambdaSignature {
+ type_signature,
+ volatility,
+ }
+ }
+
+ /// User-defined coercion rules for the function.
+ pub fn user_defined(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::UserDefined,
+ volatility,
+ }
+ }
+
+ /// An arbitrary number of lambdas or arguments of any type.
+ pub fn variadic_any(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::VariadicAny,
+ volatility,
+ }
+ }
+
+ /// A specified number of arguments of any type
+ pub fn any(arg_count: usize, volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::Any(arg_count),
+ volatility,
+ }
+ }
+}
+
+impl PartialEq for dyn LambdaUDF {
+ fn eq(&self, other: &Self) -> bool {
+ self.dyn_eq(other.as_any())
+ }
+}
+
+impl PartialOrd for dyn LambdaUDF {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ let mut cmp = self.name().cmp(other.name());
+ if cmp == Ordering::Equal {
+ cmp = self.signature().partial_cmp(other.signature())?;
+ }
+ if cmp == Ordering::Equal {
+ cmp = self.aliases().partial_cmp(other.aliases())?;
+ }
+ // Contract for PartialOrd and PartialEq consistency requires that
+ // a == b if and only if partial_cmp(a, b) == Some(Equal).
+ if cmp == Ordering::Equal && self != other {
+ // Functions may have other properties besides name and signature
+ // that differentiate two instances (e.g. type, or arbitrary parameters).
+ // We cannot return Some(Equal) in such case.
+ return None;
+ }
+ debug_assert!(
+ cmp == Ordering::Equal || self != other,
+ "Detected incorrect implementation of PartialEq when comparing functions: '{}' and '{}'. \
+ The functions compare as equal, but they are not equal based on general properties that \
+ the PartialOrd implementation observes,",
+ self.name(),
+ other.name()
+ );
+ Some(cmp)
+ }
+}
+
+impl Eq for dyn LambdaUDF {}
+
+impl Hash for dyn LambdaUDF {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.dyn_hash(state)
+ }
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+#[derive(Debug, Clone)]
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments and lambdas to the function
+ pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>,
+ /// Field associated with each arg, if it exists
+ /// For lambdas, it will be the field of the result of
+ /// the lambda if evaluated with the parameters
+ /// returned from [`LambdaUDF::lambdas_parameters`]
+ pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned
+ /// (from `return_field_from_args`) when creating the
+ /// physical expression from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+impl LambdaFunctionArgs {
+ /// The return type of the function. See [`Self::return_field`] for more
+ /// details.
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+}
+
+/// A lambda argument to a LambdaFunction
+#[derive(Clone, Debug)]
+pub struct LambdaArgument {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ body: Arc<dyn PhysicalExpr>,
+}
+
+impl LambdaArgument {
+ pub fn new(params: Vec<FieldRef>, body: Arc<dyn PhysicalExpr>) -> Self {
+ Self { params, body }
+ }
+
+ /// Evaluate this lambda
+ /// `args` should evaluate to the value of each parameter
+ /// of the correspondent lambda returned in [LambdaUDF::lambdas_parameters].
+ pub fn evaluate(
+ &self,
+ args: &[&dyn Fn() -> Result<ArrayRef>],
+ ) -> Result<ColumnarValue> {
+ let columns = args
+ .iter()
+ .take(self.params.len())
+ .map(|arg| arg())
+ .collect::<Result<_>>()?;
+
+ let schema = Arc::new(Schema::new(self.params.clone()));
+
+ let batch = RecordBatch::try_new(schema, columns)?;
+
+ self.body.evaluate(&batch)
+ }
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+#[derive(Clone, Debug)]
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field of the result of the
+ /// lambda if evaluated with the parameters returned from [`LambdaUDF::lambdas_parameters`]
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ /// ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)),
+ /// ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false))
+ /// ]`
+ pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `array_transform([1], v -> v == 5)`
+ /// this field will be `[Some(ScalarValue::List(...), None]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// An argument to a lambda function
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum ValueOrLambda<V, L> {
+ /// A value with associated data
+ Value(V),
+ /// A lambda with associated data
+ Lambda(L),
+}
+
+/// Trait for implementing user defined lambda functions.
+///
+/// This trait exposes the full API for implementing user defined functions and
+/// can be used to implement any function.
+///
+/// See [`array_transform.rs`] for a commented complete implementation
+///
+/// [`array_transform.rs`]: https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/array_transform.rs
+pub trait LambdaUDF: Debug + DynEq + DynHash + Send + Sync {
+ /// Returns this object as an [`Any`] trait object
+ fn as_any(&self) -> &dyn Any;
+
+ /// Returns this function's name
+ fn name(&self) -> &str;
+
+ /// Returns any aliases (alternate names) for this function.
+ ///
+ /// Aliases can be used to invoke the same function using different names.
+ /// For example in some databases `now()` and `current_timestamp()` are
+ /// aliases for the same function. This behavior can be obtained by
+ /// returning `current_timestamp` as an alias for the `now` function.
+ ///
+ /// Note: `aliases` should only include names other than [`Self::name`].
+ /// Defaults to `[]` (no aliases)
+ fn aliases(&self) -> &[String] {
+ &[]
+ }
+
+ /// Returns the name of the column this expression would create
+ ///
+ /// See [`Expr::schema_name`] for details
+ fn schema_name(&self, args: &[Expr]) -> Result<String> {
+ Ok(format!(
+ "{}({})",
+ self.name(),
+ schema_name_from_exprs_comma_separated_without_space(args)?
+ ))
+ }
+
+ /// Returns a [`LambdaSignature`] describing the argument types for which this
+ /// function has an implementation, and the function's [`Volatility`].
+ ///
+ /// See [`LambdaSignature`] for more details on argument type handling
+ /// and [`Self::return_field_from_args`] for computing the return type.
+ ///
+ /// [`Volatility`]: datafusion_expr_common::signature::Volatility
+ fn signature(&self) -> &LambdaSignature;
+
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2.0, 8.0], v -> v > 4.0)`
+ ///
+ /// ```ignore
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Float32, false)))]), // the Field of the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
Review Comment:
I think we no longer have `ValueOrLambdaParameter` in this PR
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
Review Comment:
Do you think this implementation can also be simplified like `coerce_value_types` to just receive &`[DataType] `instead of the enum? So it doesn't require returning `Option` and it can just return instead `Result<Vec<Vec<Field>>>` instead. I think it would also avoid the need to call `value_lambda_pair`
##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -744,6 +746,38 @@ impl TreeNodeRewriter for TypeCoercionRewriter<'_> {
});
Ok(Transformed::yes(new_expr))
}
+ Expr::LambdaFunction(LambdaFunction { func, args }) => {
+ let current_fields = args
+ .iter()
+ .map(|arg| match arg {
+ Expr::Lambda(_) => Ok(ValueOrLambda::Lambda(())),
+ _ => Ok(ValueOrLambda::Value(arg.to_field(self.schema)?.1)),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let new_fields =
+ value_fields_with_lambda_udf(¤t_fields, func.as_ref())?;
+
+ let transformed = current_fields != new_fields;
+
+ let new_args = if transformed {
Review Comment:
Is this check necessary? I see `ScalarFunction` just returns `Transformed::yes `regardless there was casting or not, also `cast_to` will only cast if types differ. I don't have a strong, this is super nit.
##########
datafusion/expr/src/type_coercion/functions.rs:
##########
@@ -148,6 +151,74 @@ pub fn fields_with_udf<F: UDFCoercionExt>(
.collect())
}
+/// Performs type coercion for lambda function arguments.
+///
+/// For value arguments, returns the field to which each
+/// argument must be coerced to match `signature`.
+/// For lambda arguments, returns a clone of the associated data
+///
+/// For more details on coercion in general, please see the
+/// [`type_coercion`](crate::type_coercion) module.
+pub fn value_fields_with_lambda_udf<L: Clone>(
+ current_fields: &[ValueOrLambda<FieldRef, L>],
+ func: &dyn LambdaUDF,
+) -> Result<Vec<ValueOrLambda<FieldRef, L>>> {
+ match func.signature().type_signature {
+ LambdaTypeSignature::UserDefined => {
+ let arg_types = current_fields
+ .iter()
+ .filter_map(|p| match p {
+ ValueOrLambda::Value(field) => Some(field.data_type().clone()),
+ ValueOrLambda::Lambda(_) => None,
+ })
+ .collect::<Vec<_>>();
+
+ let coerced_types = func.coerce_value_types(&arg_types)?;
+
+ if coerced_types.len() != arg_types.len() {
+ return plan_err!(
+ "{} coerce_value_types should have returned {} items but returned {}",
+ func.name(),
+ arg_types.len(),
+ coerced_types.len()
+ );
+ }
+
+ let mut coerced_types = coerced_types.into_iter();
+
+ Ok(current_fields
Review Comment:
Can we add a small comment here explaining that this resconstruction loop? maybe it's worth mentioning this allows support for interleaved lambdas as well
##########
datafusion/physical-expr/src/expressions/lambda.rs:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda expression: [`LambdaExpr`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+use datafusion_common::plan_err;
+use datafusion_common::{HashSet, Result, internal_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents a lambda with the given parameters names and body
+#[derive(Debug, Eq, Clone)]
+pub struct LambdaExpr {
+ params: Vec<String>,
+ body: Arc<dyn PhysicalExpr>,
+}
Review Comment:
`LambdaExpr` and `LambdaArgument` are very similar -- both have params and body, but `LambdaExpr` uses `Vec<String>` for params while `LambdaArgument` uses `Vec<FieldRef>`. The fact that `LambdaExpr::evaluate()` returns an error but `LambdaArgument` (which does not implement `PhysicalExpr`) has its own `evaluate()` suggests some redundancy. Would it be worth exploring a design where we can just use `PhysicalExpr` directly?
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,109 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ LambdaFunction(LambdaFunction),
+ /// A Lambda expression with a set of parameters names and a body
+ Lambda(Lambda),
+ /// A named reference to a lambda parameter
+ LambdaVariable(LambdaVariable),
Review Comment:
At first It was a bit hard to picture how these three fit together, should we add some small diagram here to explain with some examples?
for example `array_transform([1,2,3], v -> v+1)` would be equivalent to:
```
LambdaFunction(array_transform)
├── args[0]: Literal([1,2,3])
└── args[1]: Lambda
└── body: BinaryExpr(+)
├── LambdaVariable("v")
└── Literal(1)
```
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ let (list, _lambda) = value_lambda_pair(self.name(), args)?;
+
+ let field = match list.data_type() {
+ DataType::List(field) => field,
+ DataType::LargeList(field) => field,
+ DataType::FixedSizeList(field, _) => field,
Review Comment:
From @rluvaton comment [here](https://github.com/apache/datafusion/pull/18921#issuecomment-3944235156) I assume we want to support ListView/LargeListView/Map at some point, we should probably include it that in a feature checklist in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2966475263
##########
datafusion/physical-expr/src/expressions/lambda.rs:
##########
@@ -0,0 +1,136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda expression: [`LambdaExpr`]
+
+use std::hash::Hash;
+use std::sync::Arc;
+use std::{any::Any, sync::OnceLock};
+
+use crate::expressions::Column;
+use crate::physical_expr::PhysicalExpr;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
+use datafusion_common::{internal_err, HashSet, Result};
+use datafusion_expr::ColumnarValue;
+
+/// Represents a lambda with the given parameters names and body
+#[derive(Debug, Eq, Clone)]
+pub struct LambdaExpr {
+ params: Vec<String>,
+ body: Arc<dyn PhysicalExpr>,
+ captures: OnceLock<HashSet<usize>>,
+}
+
+// Manually derive PartialEq and Hash to work around https://github.com/rust-lang/rust/issues/78808 [https://github.com/apache/datafusion/issues/13196]
+impl PartialEq for LambdaExpr {
+ fn eq(&self, other: &Self) -> bool {
+ self.params.eq(&other.params) && self.body.eq(&other.body)
+ }
+}
+
+impl Hash for LambdaExpr {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.params.hash(state);
+ self.body.hash(state);
+ }
+}
+
+impl LambdaExpr {
+ /// Create a new lambda expression with the given parameters and body
+ pub fn new(params: Vec<String>, body: Arc<dyn PhysicalExpr>) -> Self {
+ Self {
+ params,
+ body,
+ captures: OnceLock::new(),
+ }
+ }
+
+ /// Get the lambda's params names
+ pub fn params(&self) -> &[String] {
+ &self.params
+ }
+
+ /// Get the lambda's body
+ pub fn body(&self) -> &Arc<dyn PhysicalExpr> {
+ &self.body
+ }
+
+ pub fn captures(&self) -> &HashSet<usize> {
+ self.captures.get_or_init(|| {
Review Comment:
> But we can also remove it from creation and rerun for every batch, just let me know!
This code is not in this PR anymore I don't remember how exactly it ended up looking :D but my main point was that `OnceLock` was a bit overkill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2902626011
##########
datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:
##########
Review Comment:
Yes, thanks https://github.com/apache/datafusion/pull/18921/commits/83dfbdd13e98be8a00f90becc26f5b9f277eebff
I believe we are going to produce a regular `ScalarFunction` for `LambdaFunction` nodes, so I added the note where scalar functions are handled
The unimplemented notes for `Lambda` and `LambdaParameterReference` are added when upgrading substrait-rs on #20750:
https://github.com/apache/datafusion/pull/20750/changes#diff-a6f9b212a27eb5019705752e4a035c4fa8e8c55b60c95326660f38ae839e9ad2R59-R61
https://github.com/apache/datafusion/pull/20750/changes#diff-eb46b65774adea5cafd3039875348693a50015639ec21d0409a5e2acdbf74eaeR96-R98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2896741915
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
nice, I agree the substrait consuming producer can be left as a follow up, should we bump substrait in datafusion in the meantime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4134019219
Thanks @liaCastaneda, that's great. I created the EPIC for now at #21172, and will be creating the most important drafts on the next days. And yes, I opened this for review, finally :rocket:
cc @comphead this is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3085155465
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
I agree with @rluvaton, I think it's ok to assume the wrapper will report the wrapped expression as a child -- otherwise there is no way of finding the lambda 🤷♀️ (besides, children() is mandatory, which iiuc usually means that if it's not implemented correctly there is no guarantee of correctness). We can just traverse children() until we find the lambda.
Regarding your initial comment @rluvaton -- `args` already contains the lambda as the second argument, as @gstvg says:
```
args[0] = LiteralExpr([1, 2]) ← value array
args[1] = LambdaExpr(v -> v*2) ← lambda
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3081123480
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
I pinged you there https://github.com/apache/datafusion/pull/18921/changes#r3081130002
link to my fork: https://github.com/gstvg/arrow-datafusion/blob/5c0b41df235a28ae508caad58b160b4c008b1d7d/datafusion/physical-expr/src/higher_order_function.rs#L245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3979161162
@comphead Very cool, the LambdaVariable I'm using here was inspired by Spark btw, both ended up looking very similar. But it seems to support only a single lambda parameter?
I updated the PR description and also pushed DOC.md in case you have any comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3081589663
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
the wrapper will have to report the wrapped expression as a child
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4236973430
Note, this is a breaking change since it add to expr enums more variants (we should make that enum non exhaustive so these kind of prs will not cause breaking change.
updated description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073449334
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
+ index: usize,
+ field: FieldRef,
+}
+
+impl Eq for LambdaVariable {}
+
+impl PartialEq for LambdaVariable {
+ fn eq(&self, other: &Self) -> bool {
+ self.name == other.name && self.field == other.field
+ }
+}
+
+impl Hash for LambdaVariable {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name.hash(state);
+ self.field.hash(state);
+ }
+}
+
+impl LambdaVariable {
+ /// Create a new lambda variable expression
+ pub fn new(name: String, index: usize, field: FieldRef) -> Self {
+ Self { name, index, field }
+ }
+
+ /// Get the variable's name
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Get the variable's index
+ pub fn index(&self) -> usize {
+ self.index
+ }
+
+ /// Get the variable's field
+ pub fn field(&self) -> &FieldRef {
+ &self.field
+ }
+}
+
+impl std::fmt::Display for LambdaVariable {
+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+ write!(f, "{}@", self.name)
+ }
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ if self.index >= batch.num_columns() {
+ return internal_err!(
+ "PhysicalExpr LambdaVariable references column '{}' at index {} (zero-based) but batch only has {} columns: {:?}",
+ self.name,
+ self.index,
+ batch.num_columns(),
+ batch
+ .schema_ref()
+ .fields()
+ .iter()
+ .map(|f| f.name())
+ .collect::<Vec<_>>()
+ );
+ }
+
+ Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ vec![]
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ _children: Vec<Arc<dyn PhysicalExpr>>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ Ok(self)
+ }
+
+ fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{}", self.name)
+ }
+}
+
+/// Create a lambda variable expression
+pub fn lambda_variable(
+ name: impl Into<String>,
+ field: FieldRef,
+ schema: &Schema,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let name = name.into();
+ let index = schema.index_of(&name)?;
+
+ let schema_field = schema.field(index);
+
+ if field.as_ref() != schema_field {
+ return plan_err!(
+ "LambdaVariable owned field differ from schema field {field} != {schema_field}"
+ );
+ }
Review Comment:
If you can get the field from the schema, why pass the field in the first place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074605176
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
The lambda expression? It's included in args wrapped in a `LambdaExpr` with other normal value args. It's okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074564976
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
Review Comment:
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3002224991
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
🤔 This is a good point I was not aware of. It's tricky because in `array_transform` this could be a problem or not depending on which expression it gets (if it's a multiplication it's not an issue but if it's a division it is), so I imagine the best option here is to be conservative and always clean up.
Although, we do have cheap ways to know if arrow Arrays have nulls (`array.null_count()` etc), so we could skip the cleanup entirely when there are no nulls. I guess this cleanup would be the same for all lambda functions, so it probably makes sense to handle it transparently in datafusion before calling the user's invoke? That way implementors never have to think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2999288744
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
Review Comment:
Yep. New methods like `positional_coerce_value_types`/`positional_lambdas_parameters` with default implementations that would just call `coerce_value_types`/`lambdas_parameters` just like `ScalarUDFImpl` `return_field`/`return_field_from_args` could detect this without complicating things for simpler lambda UDFs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2626020310
##########
datafusion/expr/src/expr.rs:
##########
@@ -398,6 +398,30 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Lambda expression
+ Lambda(Lambda),
+ LambdaColumn(LambdaColumn),
Review Comment:
@gstvg Would `LambdaVariable` be a more accurate name here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3869546935
@linhr
Really really sorry for the delay
<br>
> I think it's possible to resolve lambda parameters and remove them from the arguments list with `ScalarUDF` itself, with a few cons compared to a dedicated to `LambdaUDF`, but with much less code changes and similar enough to compare the alternatives. I believe I'm already finishing it, and will push to another branch soon.
Unfortunately, trying to resolve lambdas with `ScalarUDF` didn't work well.
So, before moving to a `LambdaUDF` based implementation, I would like to further discuss the current approach.
<br>
>`Expr::Lambda` and `Expr::LambdaColumn` are only fragments whose data types etc. are not well-defined which may be challenging to work with in other parts of the library (e.g. `ExprSchemable`).
>>Yeah, this is the major challenge in this PR. Currently, `Expr::Lambda` always return `DataType::Null` and `Expr::LambdaVariable` embeds a `FieldRef` which is used to implement the `ExprSchemable` and don't even look at the `DFSchema`
I mean, this *was* the major challenge of the PR on it's first implementation, now that we use `LambdaVariable` with a `Field`, I consider this to be solved. And currently `Expr::Lambda` actually returns the `return_field` of it's body, but I believe we can also return a field with `DataType::Null` if deemed better.
<br>
> `LambdaFunction` can represent a "resolved" lambda function call in the logical plan. In contrast, `Expr::Lambda` and `Expr::LambdaColumn` are only fragments whose data types etc. are not well-defined which may be challenging to work with in other parts of the library (e.g. `ExprSchemable`).
... when the function is actually "invoked" with Arrow arrays, the lambda parameter should have been resolved and removed from the argument list
Sorry, at first I thought this only means resolving/embedding the lambda body within the UDF itself and removing it from `Expr::ScalarFunction.args`/`LambdaFunction.args`, and also removing `Expr::Lambda` variant from the logical Expr enum and it's physical counterpart, but now I'm not sure it also means omitting the lambda body from `TreeNode` traversals?
To streamline the discussion, I have a few comments for each of these options, if they're applicable:
## Omitting the lambda body from TreeNode traversals
At least the following traversals currently use `TreeNode` and would require adjust if the body is omitted:
Projection pushdown for column capture, type coercion for correctness, expr simplifier and CSE for performance.
Being only 4 internal traversals, is easy to specially handle them, but since DF is an open system, I believe there should be a way to downstream users to visit/transform lambdas expressions. We could document that they need to be specially handled, outside `TreeNode`, or we could offer an API for it, and after all, I believe the ideal API would be exactly the `TreeNode` API.
In my first iteration of this PR, the "outdated" sections of the description and of the first comment, I manually checked every `TreeNode` usage on logical and physical expressions, and find none where `Expr::Lambda` should be specially handled(it is simply ignored), and `Expr::LambdaVariable` only required handling in few places.
## Removing Expr::Lambda variant from Expr enum and it's physical counterpart
Removing `Expr::Lambda` from AST makes some tree traversals more difficult, while keeping it doesn't make difference: it's simply ignored as most expressions in most traversals.
One example is `BindLambdaVariable` in this PR. Is not important to understand what it does, just that it's bigger, with more boilerplate and harder to read without `Expr::Lambda`:
```rust
struct BindLambdaVariable<'a> {
variables: HashMap<&'a str, (ArrayRef, usize)>,
//(variable value, number of times it has been shadowed by other lambdas variables in inner scopes in the current position of the tree)
}
impl TreeNodeRewriter for BindLambdaVariable<'_> {
type Node = Arc<dyn PhysicalExpr>;
fn f_down(&mut self, node: Self::Node) -> Result<Transformed<Self::Node>> {
if let Some(lambda_variable) = node.as_any().downcast_ref::<LambdaVariable>() {
if let Some((value, shadows)) = self.variables.get(lambda_variable.name()) {
if *shadows == 0 {
return Ok(Transformed::yes(Arc::new(
lambda_variable.clone().with_value(value.clone()),
)));
}
}
} else if let Some(inner_lambda) = node.as_any().downcast_ref::<LambdaExpr>() {
for param in inner_lambda.params() {
if let Some((_value, shadows)) = self.variables.get_mut(param.as_str()) {
*shadows += 1;
}
}
}
Ok(Transformed::no(node))
}
fn f_up(&mut self, node: Self::Node) -> Result<Transformed<Self::Node>> {
if let Some(inner_lambda) = node.as_any().downcast_ref::<LambdaExpr>() {
for param in inner_lambda.params() {
if let Some((_value, shadows)) = self.variables.get_mut(param.as_str()) {
*shadows -= 1;
}
}
}
Ok(Transformed::no(node))
}
}
```
Implementation without Physical Lambda on AST
```rust
fn bind_lambda_variables(
node: Arc<dyn PhysicalExpr>,
params: &mut HashMap<&str, (ArrayRef, usize)>,
) -> Result<Transformed<Arc<dyn PhysicalExpr>>> {
node.transform_down(|node| {
if let Some(lambda_variable) = node.as_any().downcast_ref::<LambdaVariable>() {
if let Some((value, shadows)) = params.get(lambda_variable.name()) {
if *shadows == 0 {
return Ok(Transformed::yes(Arc::new(
lambda_variable.clone().with_value(value.clone()),
)));
}
}
} else if let Some(fun) = node.as_any().downcast_ref::<LambdaUDFExpr>() {
let mut transformed = false;
let new_lambdas = fun
.lambdas()
.iter()
.map(|(params_names, body)| {
for param in *params_names {
if let Some((_value, shadows)) = params.get_mut(param.as_str()) {
*shadows += 1;
}
}
let new_body = bind_lambda_variables(Arc::clone(body), params)?;
transformed |= new_body.transformed;
for param in *params_names {
if let Some((_value, shadows)) = params.get_mut(param.as_str()) {
*shadows -= 1;
}
}
Ok((params_names.to_vec(), new_body.data))
})
.collect::<Result<_>>()?;
let new_args = fun
.args()
.iter()
.map(|arg| {
let new_arg = bind_lambda_variables(Arc::clone(arg), params)?;
transformed |= new_arg.transformed;
Ok(new_arg.data)
})
.collect::<Result<_>>()?;
let fun = fun.with_new_children(new_args, new_lambdas);
return Ok(Transformed::new(
Arc::new(fun),
transformed,
TreeNodeRecursion::Stop,
));
}
Ok(Transformed::no(node))
})
}
```
Note that this transformation always return a constant `TreeNodeRecursion` value. If not, `TreeNodeRecursion` would also needed to be manually handled, adding more boilerplate that `TreeNode` handles automatically when `Expr::Lambda` is present
## Resolve/embed lambdas on the UDF implementation itself(essentially partition args from lambdas)
By partition args and lambdas, we lose positional data, which are necessary to implement `SqlUnparser` and logical and physical formatting, leading to either the implementation itself having to implement them, adding some boilerplate, where today is automatically handled by `ScalarUDF` and `SqlUnparser` itself, or that the implementation save and expose positional data, that is them used by `ScalarUDF` and `SqlUnparser` to de-partition the args list, and them proceed normally.
Some closed system without support for udf with lambdas use a common positioning for lambdas and can just use them without having to store positional data nor lambdas and other args in the same ordered list:
Always last for spark, duckdb and snowflake:
```sql
transform(array(1), v -> v*2) -- spark
list_transform([1, 2, NULL, 3], lambda x: x + 1) -- duckdb
transform([1, 2, 3], a INT -> a * 2) -- snowflake
```
Always first for clickhouse:
```sql
arrayMap(x -> (x + 2), [1, 2, 3])
```
But I think that datafusion as an open and extendable system shouldn't stick to a single convention as different deployments may want to support one, another or both conventions.
Furthermore, my interest in lambda functions is to implement union manipulating udfs, which receives multiple lambdas interleaved with regular arguments. Consider an `Union<'str' = Utf8, 'num' = Float64>`, that could be transformed like this:
```sql
union_transform(
union_value,
'str', str -> trim(str),
'num', num -> num*2
)
```
where today the only way is with this(if/when support for `union_value` lands):
```sql
case union_tag(union_value)
when 'str' then union_value('str', trim(union_extract(union_value, 'str')))
when 'num' then union_value('num', union_extract(union_value, 'num') * 2)
end
```
For such cases, storing positional data or manually implementing logical and phyiscal formatting and `SqlUnparser` for every UDF is mandatory, as that positioning doesn't fit the simple conventions like always first or last.
This also omit positional info from the implementation. None lambda udf discussed so far, including `union_transform`, requires positional info for evaluation, only for logical and physical formatting and `SqlUnparser`, but I don't like the idea of blocking any future hypothetical udf that requires positional info for evaluation. A theoretical example is `union_transform` itself supporting a slight shorter syntax for lambdas that returns constants:
```sql
union_transform(
union_value,
'str', str -> trim(str),
'num', _num -> 0 -- lambda that returns constant/scalar
)
union_transform(
union_value,
'str', str -> trim(str),
'num', 0 -- shorter syntax, pass constant directly
)
```
That syntax does requires positional info for evaluation. It's really just an example and I don't plan to support it on my `union_transform` implementation
Instead, if we add a new `LambdaFunction` expression and `LambdaUDF` trait, the implementation could own both the lambdas *and* the other args, so that no partitioning would occur:
```rust
struct LambdaFunction { invocation: Box<dyn LambdaInvocation> } //invocation includes all args, both lambdas and non-lambdas
Expr::LambdaFunction(LambdaFuntion::new(ArrayTransform::new(all_args_including_lambdas)))
// instead of
struct LambdaFunction { func: Arc<dyn LambdaUDF>, args: Vec<Expr> } // func owns only lambdas, and non lambdas are stored on args
Expr::LambdaFunction(LambdaFunction::new(ArrayTransform::new(lambda), args_without_lambdas))
```
# Insights from Spark implementation
I'm don't have much experience with Spark and Scala, I only implemented a custom data source a few years ago, so if something below seems wrong, it probably is.
Spark includes both [HigherOrderFunction trait](https://github.com/apache/spark/blob/eec092c9f9d1ad3df0390b1ea56340c1103ba887/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L153) and [LambdaFunction class](https://github.com/apache/spark/blob/eec092c9f9d1ad3df0390b1ea56340c1103ba887/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L113), similar to our current `Expr::Lambda` (and not the proposed `LambdaFunction`), as well as [NamedLambdaVariable class](https://github.com/apache/spark/blob/eec092c9f9d1ad3df0390b1ea56340c1103ba887/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77), similar to ours `Expr::LambdaVariable`, all of them extending [Expression](http://javadoc.io/static/org.apache.spark/spark-catalyst_2.10/2.2.3/index.html#org.apache.spark.sql.catalyst.expressions.Expression), which in turn extends [TreeNode](https://www.java
doc.io/static/org.apache.spark/spark-catalyst_2.10/2.2.3/index.html#org.apache.spark.sql.catalyst.trees.TreeNode)
Some traversals branch directly on `LambdaFunction` expressions, which in my view support the idea of both keeping `Expr::Lambda` and exposing the lambda body on the `TreeNode` API:
[Lambda Binder](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/LambdaBinder.scala#L56), (similar to ours BindLambdaVariable)
[ResolveLambdaVariables](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala#L59)
[ColumnResolutionHelper](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala#L121)
[ColumnNodeToExpressionConverter](https://github.com/apache/spark/blob/f80b919d19ce8e390c029857802cfe40190bb5ea/sql/core/src/main/scala/org/apache/spark/sql/classic/columnNodeSupport.scala#L143)
[CheckAnalysis](https://github.com/apache/spark/blob/f80b919d19ce8e390c029857802cfe40190bb5ea/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L352)
[ColumnNodeToProtoConverter](https://github.com/apache/spark/blob/f80b919d19ce8e390c029857802cfe40190bb5ea/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/columnNodeSupport.scala#L164)
Some on branch `LambdaVariable`, which in my view support the ideia exposing of the lambda body on the `TreeNode` API:
[SessionCatalog](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L1641)
[HigherOrderFunction.functionForEval](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L221)
[NormalizePlan](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/NormalizePlan.scala#L107)
[TableOutputResolver](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala#L265)
Some branch on `HigherOrderFunction`:
[CheckAnalysis](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L355)
[Analyzer](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2168)
[ResolveLambdaVariables](https://github.com/apache/spark/blob/620b2f65f0daed19e1800503df3c892b3bb664e1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala#L55)
Is true that some traversals branch on `HigherOrderFunction` expressions, therefore supporting the ideia of a new `Expr::LambdaFunction`, but since checking if a `Expr::ScalarFunction` invocation contains lambdas is simple as the code below, and can be made even simpler by adding a helper method `contains_lambdas(&self) -> bool` to `ScalarFunction`, I don't think it alone justifies adding a new `Expr::LambdaFunction` holding a `ScalarFunction`. If `LambdaUDF` trait is added then a new `Expr::LambdaFunction` variant has to be added anyway so this become a non-issue.
```rust
match expr {
Expr::ScalarFunction(fun) if fun.args.iter().any(|arg| matches!(arg, Expr::Lambda(_))) => ...,
// with helper
Expr::ScalarFunction(fun) if fun.contains_lambdas() => ...
...
}
impl ScalarFunction {
fn contains_lambdas(&self) -> bool {
self.args.iter().any(|arg| matches!(arg, Expr::Lambda(_)))
}
// Other helpers could be added to like:
fn lambda_args(&self) -> impl Iterator<Item = &LambdaFunction> {
self.args.iter().filter_map(|arg| match arg {
Expr::Lambda(l) => Some(l),
_ => None
})
}
fn non_lambda_args(&self) -> impl Iterator<Item = &Expr> {
self.args.iter().filter(|arg| !matches(arg, Expr::Lambda(_)))
}
}
```
[Higher order functions, like ArrayTransform, receive both the regular arg as well as the lambda as regular Expressions, the lambda being expected to be a LambdaFunction](https://github.com/apache/spark/blob/214bf958757c804e2ae637b3fc3d4ca9ec94a1d1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L312-L328):
```scala
case class ArrayTransform(
argument: Expression,
function: Expression)
extends ArrayBasedSimpleHigherOrderFunction with CodegenFallback {
override def dataType: ArrayType = ArrayType(function.dataType, function.nullable)
override protected def bindInternal(
f: (Expression, Seq[(DataType, Boolean)]) => LambdaFunction): ArrayTransform = {
val ArrayType(elementType, containsNull) = argument.dataType
function match {
case LambdaFunction(_, arguments, _) if arguments.size == 2 =>
copy(function = f(function, (elementType, containsNull) :: (IntegerType, false) :: Nil))
case _ =>
copy(function = f(function, (elementType, containsNull) :: Nil))
}
}
```
[And it's instantiated like that](https://github.com/apache/spark/blob/3298e685a9641c3e137917f75f58bc0bcfdfe32e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InsertMapSortExpression.scala#L152-L155):
```scala
val param = NamedLambdaVariable("x", et, containsNull)
val funcBody = insertMapSortRecursively(param)
ArrayTransform(e, LambdaFunction(funcBody, Seq(param)))
```
So, regarding to resolving lambdas/partition lambdas from args, all args are owned by the implementation, and not by `HigherOrderFunction`, which is just a trait, therefore the lambas are indeed resolved, but so are the other args, and no partitioning occurs. I believe that our current approach would look like this in Spark:
`trait Expression`, implemented by `class HigherOrderFunction` which owns a `trait HigherOrderFunctionImpl` which is implemented by `ArrayTransform`, where the lambda and the arg would be stored in the `class HigherOrderFunction`, not in `ArrayTransform`. The Spark approach implemented in DF would be look the new `LambdaFunction` expression and `LambdaUDF` trait cited above, where implementation owns both the lambdas *and* the other args: `Expr::LambdaFunction(LambdaFuntion::new(ArrayTransform::new(all_args_including_lambdas)))`, instead of `Expr::LambdaFunction(LambdaFunction::new(ArrayTransform::new(lambdas), args_without_lambdas))`
Finally, running this script on `spark-shell` returns the following output, showing both that lambda bodies are present in `TreeNode` traversals as well as `LambdaFunction` nodes:
```scala
val df = spark.sql("SELECT transform(array(1), v -> v*2)")
val transform = df.queryExecution.logical.expressions(0)
transform.foreach { node =>
println(s"${node.nodeName}: ${node.toString}")
}
println(transform.treeString)
```
```
UnresolvedAlias: unresolvedalias(transform(array(1), lambdafunction((v * 2), v)))
UnresolvedFunction: transform(array(1), lambdafunction((v * 2), v))
UnresolvedFunction: array(1)
Literal: 1
LambdaFunction: lambdafunction((v * 2), v)
Multiply: (v * 2)
UnresolvedNamedLambdaVariable: v
Literal: 2
UnresolvedNamedLambdaVariable: v
unresolvedalias('transform('array(1), lambdafunction((lambda 'v * 2), lambda 'v, false)))
+- 'transform('array(1), lambdafunction((lambda 'v * 2), lambda 'v, false))
:- 'array(1)
: +- 1
+- lambdafunction((lambda 'v * 2), lambda 'v, false)
:- (lambda 'v * 2)
: :- lambda 'v
: +- 2
+- lambda 'v
```
In short, if we more or less agree with my points above, adding a new `LambdaUDF` just to not modify `ScalarFunctionArgs`, IMHO, doesn't seem worth the additional code that get's to be reviewed and maintained both in this PR (which is already big) and in subsequent PRs adding more lambdas UDFs like `array_filter` and `array_fold`
What do you think? I'm missing or misunderstanding something? Thanks!
And again, sorry for the delay
cc @keen85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2868461522
##########
DOC.md:
##########
Review Comment:
@comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3938399669
Thanks @linhr! It most applies to having lambdas and args partitioned, omitting the body on `TreeNode` and removing `Expr::Lambda`. Changing the PR to just use a new `Expr::LambdaFunction(Arc<dyn LambdaUDF>)` where args are unpartitioned, body is exposed on `TreeNode` and `Expr::Lambda` is kept is okay, it wouldn't incur in the other cited points, but would make the PR bigger.
Thanks @comphead! The current representation looks somewhat similar to your suggestion, except that is a `Expr::ScalarFunction`/`ScalarFunctionExpr` instead of dedicated `LambdaFunction`, and that references to lambda parameters uses `LambdaVariable` instead of `Column`, containing an optional value that should be binded before execution:
```
FunctionExpression //regular ScalarFunction
name: list_filter
children:
1. ListExpression [1,2,3,4]
2. LambdaExpression // the added physical LambdaExpr / logical Expr::Lambda
parameters: ["x"]
body:
ComparisonExpression (=)
left:
ArithmeticExpression (%)
left: LambdaVariable("x", Field::new("", Int32, false), None) that after binding becomes
LambdaVariable("x", Field::new("", Int32, false), Some([1, 2, 3, 4]))
right: Constant(2)
right:
Constant(0)
```
The reason why adding `LambdaVariable` is easier than trying to use `Column` is show on the outdated sections on this PR description and on the first comment.
If the added code for a `LambdaUDF` is considered worthwhile to not modify `ScalarUDF` and/or because it's a better representation, while keeping others things unchanged, I'm okay with it. But since you mentioned vectorized execution, I just want to make sure that we are not moving to `LambdaUDF` *only* for performance because, contrary to Spark, where the `LambdaVariable` value is set and re-set for every row, which would obviously be terrible for a vectorized engine, here it is only set once per batch, and so can perform as fast as regular expressions. The `LambdaVariable` value being a `ColumnarValue` is just a niche optimization for scalar evaluation, we can change it to support `ArrayRef` only. Native performance has been a goal since the beginning, and a `list_filter` implementation based on this can be easily vectorized. As an example, #17220 got vectorized execution while also based on `ScalarUDF` and regular physical expressions, [the lambda body is evaluated *once per
batch* and it's output used to filter the list values with the `filter` kernel, and then the offsets are adjusted:](http://github.com/apache/datafusion/pull/17220/changes#diff-3473f25bb419d339904b4f82e9e4485637dcf2b865e29c7f91ce7d646194f780R357-R387)
```rust
let filter_array = lambda.evaluate(&batch)?;
let ColumnarValue::Array(filter_array) = filter_array else {
return exec_err!("array_filter requires a lambda that returns an array of booleans");
};
let filter_array = as_boolean_array(&filter_array)?;
let filtered = filter(&values, filter_array)?;
for row_index in 0..list_array.len() {
if list_array.is_null(row_index) {
// Handle null arrays by keeping the offset unchanged
offsets.push_length(0);
continue;
}
let start = value_offsets[row_index];
let end = value_offsets[row_index + 1];
let num_true = filter_array
.slice(start.as_usize(), (end - start).as_usize())
.true_count();
offsets.push_length(num_true);
}
let offsets = offsets.finish();
let list_array = GenericListArray::<OffsetSize>::try_new(
Arc::clone(field),
offsets,
filtered,
nulls.cloned(),
)?;
Ok(Arc::new(list_array))
```
One thing that can be optimized is use [Buffer::count_set_bits_offset](https://docs.rs/arrow/latest/arrow/buffer/struct.Buffer.html#method.count_set_bits_offset) instead of `BooleanArray::slice` and then `true_count` to avoid 1-2 `Arc::clone` per loop iteration. There's also a faster way to adjust the offsets when the average sub-list length is small, something like 64 or less, but it's bigger and a bit more intricate, so I choose implement `array_transform` instead to make the PR smaller. A reasonable fast `array_fold` is even more complicated, but still implementable in any lambda approach discussed here.
`array_transform` here is also vectorized: the transforming lambda body is evaluated only once per batch/invoke/evaluate call.
<br>
> Then we likely need to Bind x to know what is it and finally rewrite it into something computable, perhaps using UDF of (Int) => Boolean
I believe that by using `LambdaVariable`, planning the lambda body into a physical expr as usual is enough to compute it, without having to use UDFs. One thing that we must take into account is column capture, which is already fully supported here, including shadowing and projection pushdown(columns from the input plan that only appear within a lambda body). As an example, #17220 does not support that, and if tried to support it while using `Expr::Column` for lambda variables and passing those variables in the `RecordBatch` to `PhysicalExpr::evaluate`, would likely get the same problems I got in my first approach, which are also show in the outdated sections of the description and of [the first comment here](https://github.com/apache/datafusion/pull/18921#issuecomment-3573747241). That's why `LambdaVariable` exists. Supporting lambdas without capture support is relatively easy, regardless of the approach.
@rluvaton I see you also [were not fond of adding physical expr's to `ScalarFunctionArgs` on #17220](https://github.com/apache/datafusion/pull/17220#discussion_r2297665061). Any comments here?
<br>
Finally, how would the new `LambdaUDF` trait works?
I believe it should look like `ScalarFunction`?
```rust
struct LambdaFunction {
args: Vec<Expr>,
fun: Arc<dyn LambdaUDF>,
}
enum Expr {
...
Expr::LambdaFunction(LambdaFunction),
}
```
It also can be made more like Spark:
```rust
trait LambdaInvoker {
...
fn invoke(&self, args: Vec<Expr>) -> Box<dyn LambdaInvocation>;
}
enum Expr {
...
Expr::LambdaFunction(Box<dyn LambdaInvocation>),
}
```
I believe `LambdaUDF`/`LambdaInvocation` trait would look like `ScalarUDFImpl` except for the `lambdas_parameters` method, `ReturnFieldArgs` in `return_field_from_args` and `LambdaFunctionArgs` in the `invoke` method, or any of you imagine more fundamental changes?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "benbellick (via GitHub)" <gi...@apache.org>.
benbellick commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2896534107
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
[I have done the release](https://github.com/substrait-io/substrait-rs/releases/tag/v0.62.3). `v0.62.3` contains the protos for lambdas.
Considering the size of this PR, I think it is appropriate to leave the handling of substrait lambdas in datafusion for a subsequent PR. But let's definitely make sure that we do this PR with the substrait interaction in mind :)
I will review the relevant parts for that reason. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3658471081
Hi @fbx31, reduce is supported too. It isn't included here because it's implementation is not as simple as array_transform and would make this PR even bigger. But if any reviewer/maintainer request it we can add it and make sure it works since the beginning.
[Nesting is supported, some tests already cover it](https://github.com/gstvg/arrow-datafusion/blob/b18d2145163fb5933dfed55eb6305412743b6cac/datafusion/sqllogictest/test_files/lambda.slt#L44-L47)
With the new approach you need to use lambda_col("current_value") instead of col("current_value"):
```rust
df.with_column(
"new_col",
array_transform(
col("my_array"),
lambda(
vec!["current_value"], // define the parameters name
lambda_col("current_value") * 2 // the lambda body
)
)
)
```
SQL equivalent:
```sql
array_transform(my_array, current_value -> current_value * 2)
```
No worries, feel free to ask any questions you have
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "linhr (via GitHub)" <gi...@apache.org>.
linhr commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3678691204
> Could you expand this further? It is like `LambdaUDF` having a `resolve_lambdas` method which result is passed to the `invoke_with_args` method?
Suppose we have `array_transform([1, 2], v -> v*2)`. We could have a trait `LambdaUDF` and have `impl LambdaUDF for ArrayTransform`. (Or we follow the existing convention to have `struct LambdaUDF` and `trait LambdaUDFImpl` separately.) A logical representation of `v -> v*2` is passed to `ArrayTransform::new()`. For `Expr::LambdaFunction(LambdaFunction)`, we can have `LambdaFunction { func: Arc<dyn LambdaUDF>, args: Vec<Expr> }` where the non-lambda parameter `[1, 2]` is stored in `args`.
During physical planning, we could resolve `ArrayTransform` into `PhysicalArrayTransform` which stores `v -> v*2` resolved as certain `PhysicalExpr`. We have a trait `PhysicalLambdaUDF` and `impl PhysicalLambdaUDF for PhysicalArrayTransform`. The trait method `PhysicalLambdaUDF::invoke_with_args` accepts the Arrow array `[1, 2]` and compute the results. I'd imagine this invocation can be done in a general (physical) `LambdaFunctionExpr` that works for all lambda functions, similar to `ScalarFunctionExpr`.
When I worked with `ScalarUDF`, I notice that the logic required for logical representation, physical planning, and the actual execution are all within a single `ScalarUDFImpl` trait. If we look at how these trait methods are used by various planning/execution stages, we might get the big picture how a parallel code structure (with multiple traits to separate the responsibilities) can be designed for lambda functions.
I haven't thought about function registry, documentation etc. which we can get for free in the existing `ScalarUDF` setup. So some more investigation is needed to estimate the amount of work if we explore the route I described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3009828234
##########
datafusion/execution/src/task.rs:
##########
Review Comment:
Should we add a public accessor here? so users can call & pass lambda functions in `TaskContext::new(...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3002322975
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Maybe a middle ground solution would be introducing a trait method with a default value so implementors can indicate whether they want DataFusion to handle this or not (or the lambda function is safe or not):
```
fn clean_null_values(&self) -> bool {
true
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4256471323
@rluvaton @gstvg WDYT for renaming this beautiful PR as WIP and create brand new with this PR code change, so people can review it properly?
I think we passed the poc/design phase discussions, so would be beneficial for folks to go through clean PR, having this as a reference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073435526
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
Review Comment:
why name if have field already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074605176
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
The lambda function/expression? It's included in args wrapped in a `LambdaExpr` with other normal value args. It's okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2908863495
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
Yes, it's possible and would work for every lambda function I know, but before doing it, there's a hypothetical kind of lambda function that may require positional data to implement coercion. Consider a `union_transform` function which transform the values of each variant independently:
```sql
union_transform(
union_value,
'str', str -> trim(str),
'num', num -> num*2
)
```
If it wanted to support receiving a scalar directly instead of a lambda that return a scalar and use `coerce_value_types`, it would require knowing the position of every argument to know if a given string is variant name or a string to be set as the value of a variant. Evaluation would not be affected by this change.
```sql
union_transform(
union_value,
'str', _str -> '<redacted>', -- lambda that returns constant/scalar
'num', num -> num*2
)
union_transform(
union_value,
'str', '<redacted>', -- shorter syntax, pass constant directly
'num', num -> num*2
)
```
This is really just a theoretical example and I don't need to support that syntax on my `union_transform` implementation.
So, is it worth supporting this or is it too hypothetical ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976030740
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ let (list, _lambda) = value_lambda_pair(self.name(), args)?;
+
+ let field = match list.data_type() {
+ DataType::List(field) => field,
+ DataType::LargeList(field) => field,
+ DataType::FixedSizeList(field, _) => field,
Review Comment:
ListView/LargeListView are currently [handled via type coercion](https://github.com/apache/datafusion/pull/18921/changes#diff-e464d894d5a0fe37392c40f9e54f67590220578259a8dc41fa57b329f6f0d343R114-R115). This makes things simpler and if the list view is casted to normal list multiple times on a projection, CSE will dedup it. I can only imagine few cases where it's more efficiently to handle views directly, but I'm not sure if they are worth handling:
1. Every value is referenced by at least one view, regardless of ordering, so we can transform them directly without packing them first, but that I guess that's uncommon and that check is a bit costly for the false cases.
2. All referenced values are packed on order. Is cheaper to check, but if that would make sense here, it likely would make sense on arrow_cast ListView->List instead, as it allow zero-copy casting, and we would get it for free here as well as CSE.
3. The sublists avg size is very high to the point is efficient to call the lamba body physical expr for every sublist instead of packing them together first, and the output type of the transformation is cheaper to move around than the inputs, because we would need to concat the transformed sublists into a single array, e.g: `array_transform(list_view_of_strings, str -> length(str))`. Instead of packing `list_view_of_strings`, which is expensive for strings, we could call the lambda body for each sublist, and then concat the resulting int32 arrays, which is cheaper than packing strings.
And for maps, I thought it would be a different function, like `map_transform(map_col, (k, v, i) -> ...)`. Shoud we support it here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3950771759
@rluvaton @comphead
I updated the PR to a `LambdaUDF` trait based implementation. It added [1300 lines](https://github.com/apache/datafusion/pull/18921/commits/e1921eb377a56766b2dd4729311def1c3515c8c4), totaling 3000, mostly boilerplate from `ScalarUDF` including a lot of documentation. Was that on the range you were expecting?
[Based on the ScalarUDF docs stating that it exists to maintain backwards compatibility with an older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note), I included only a `LambdaUDF` trait and not a `struct LambdaUDF` + `trait LambdaUDFImpl` pair to not make even bigger. There's a few small things missing that I want to implement tomorrow, and if all of you are okay with the results, I want to open this to review. My only concern is that the PR size may delay the review, as in https://github.com/apache/datafusion/pull/17220#issuecomment-3223911837, but since IMHO this PR is simpler, I hope it will be that long to review, despite the size
> 1. The ScalarUDF trait will not grow too much and make implementing regular scalar UDFs easier or lambda overwhelming
> 2. what if we need to add a required function but only for lambda, we can add it on the new trait with ease and we won't need to do some weird stuff to avoid breaking changes.
> 3. Less ambiguity on the API.
Yeah, I think that 2 is the main point. The previous ScalarUDF based approach only added a single method to the trait that already contains 20, it didn't require any change for non-lambda udfs (this would be unacceptable), and compared to the actual LambdaUDF based, lambda UDFs only required 2 additional lines of code per implementation, but, while I can't think of any ... all my previous counter-arguments can fall apart in the future with a single requirement change. There's upfront cost in review time and time to merge, but also more room to work in the future
> Support Map, (Large)List, (Large)ListView, FixedSizeList as the input for lambda
Currently any type is accepted and passed to to the implementation to derive the parameters from it, usually the inner values of a list, but I want to work with unions too, for example. But as of now array_transform doesn't handle ListView's (I thought there's no plans to support it overall?). Since the ListView values may contain unreferenced values, should it be compacted, or casted to a regular compacted List? And if so, return the transformed List or cast it to ListView?
> multiple lambdas in a single expression, for example map_key_value(some_map_col, map_key_lambda, map_value_lambda) and each lambda gets a different variables
[Supported: LambdaFunctionArgs.args can hold multiple lambdas](https://github.com/apache/datafusion/pull/18921/changes#diff-e464d894d5a0fe37392c40f9e54f67590220578259a8dc41fa57b329f6f0d343R143-R149)
```rust
fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
(list_value, lambda)
else {
return exec_err!(...
```
Since the beginning I worked to support multiple lambdas to implement union manipulating functions like this:
```sql
union_transform(
union_value,
'str', str -> trim(str),
'num', num -> num*2,
'bool', bool -> NOT bool,
)
```
> Lambda expression that access columns that are not in the list itself
[Already supported and tested (I call it column capture through the PR and comments. Please ignore the comment above the test, I'm going to remove it)
](https://github.com/apache/datafusion/pull/18921/changes#diff-53634af9d48017f60cb545dfe2aaab20d1f7fa3435947c161ef22515fa62ed8dR55-R58)
```sql
CREATE TABLE t as SELECT 1 as n;
query ?
SELECT array_transform([1, 2], (e) -> n) from t;
----
[1, 1]
```
> optional arguments for lambda, for example the index of the item in the list
the optional here is important as I want to avoid creating that input if I don't need to.
[Also supported ](https://github.com/apache/datafusion/pull/18921/changes#diff-e464d894d5a0fe37392c40f9e54f67590220578259a8dc41fa57b329f6f0d343R168-R177)
```rust
// use closures so that bind_lambda_variables evaluates only the params that are actually referenced
// avoiding unnecessary computations
let values_param = || Ok(Arc::clone(list_values));
let indices_param = || elements_indices(&list_array);
let binded_body = bind_lambda_variables(
Arc::clone(&lambda.body),
&lambda.params,
&[&values_param, &indices_param],
)?;
```
*after a refactor, bind_lambda_params actually eager evaluated all params, but I'll fix that
> Nested lambda expressions
[Supported](http://github.com/apache/datafusion/pull/18921/changes#diff-53634af9d48017f60cb545dfe2aaab20d1f7fa3435947c161ef22515fa62ed8dR45-R47)
```sql
SELECT array_transform(t.v, (v1, i) -> array_transform(v1, (v2, j) -> array_transform(v2, v3 -> j)) ) from t;
----
[[[0, 1], [0]], [[0]], [[]]]
```
> array_transform, the tricky part for this function is its return type depends on lambda
`LambdaUDF` contains a method `lambdas_arguments` where the implementation must return the type of the parameters of all it's lambdas when evaluated with a given set of values. Then DF uses this info to compute the type of the lambdas and pass it on `return_field_from_args`, so the implementation can easily compute it's return type. For example the expr `array_transform([1, 2], v -> repeat(v, 'a'))`, `lambdas_arguments` would receive `[ValueOrLambda::Value(List(Int32)), ValueOrLambda::Lambda]` and should return `vec![None, Some(vec![DataType::Int32])]`. Then `return_field_from_args` would be called with `[ValueOrLambda::Value(List(Int32)), ValueOrLambda::Lambda(Utf8)]`, where the implementation would need just to return `List(Utf8)`.
<details><summary>return_field_from_args method</summary>
```rust
pub struct LambdaReturnFieldArgs<'a> {
/// The data types of the arguments to the function
///
/// If argument `i` to the function is a lambda, it will be the field returned by the
/// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
///
/// For example, with `array_transform([1], v -> v == 5)`
/// this field will be `[Field::new("", DataType::List(DataType::Int32), false), Field::new("", DataType::Boolean, false)]`
pub arg_fields: &'a [ValueOrLambdaField],
/// Is argument `i` to the function a scalar (constant)?
///
/// If the argument `i` is not a scalar, it will be None
///
/// For example, if a function is called like `my_function(column_a, 5)`
/// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
pub scalar_arguments: &'a [Option<&'a ScalarValue>],
}
/// A tagged Field indicating whether it correspond to a value or a lambda argument
#[derive(Clone, Debug)]
pub enum ValueOrLambdaField {
/// The Field of a ColumnarValue argument
Value(FieldRef),
/// The Field of the return of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
Lambda(FieldRef),
}
fn return_field_from_args(
&self,
args: datafusion_expr::LambdaReturnFieldArgs,
) -> Result<Arc<Field>> {
let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
take_function_args(self.name(), args.arg_fields)?
else {
return exec_err!(
"{} expects a value follewed by a lambda, got {:?}",
self.name(),
args
);
};
//TODO: should metadata be copied into the transformed array?
// lambda is the resulting field of executing the lambda body
// with the parameters returned in lambdas_parameters
let field = Arc::new(Field::new(
Field::LIST_FIELD_DEFAULT_NAME,
lambda.data_type().clone(),
lambda.is_nullable(),
));
let return_type = match list.data_type() {
DataType::List(_) => DataType::List(field),
DataType::LargeList(_) => DataType::LargeList(field),
DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
_ => unreachable!(),
};
Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
}
```
</details>
<details><summary>lambdas_parameters method</summary>
```rust
trait LambdaUDF {
/// Returns the parameters that any lambda supports
fn lambdas_parameters(
&self,
args: &[ValueOrLambdaParameter],
) -> Result<Vec<Option<Vec<Field>>>>;
}
pub enum ValueOrLambdaParameter<'a> {
/// A columnar value with the given field
Value(FieldRef),
/// A lambda
Lambda,
}
// array_transform implementation
impl LambdaUDF for ArrayTransform {
fn lambdas_parameters(
&self,
args: &[ValueOrLambdaParameter],
) -> Result<Vec<Option<Vec<Field>>>> {
let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] =
args
else {
return exec_err!(
"{} expects a value followed by a lambda, got {:?}",
self.name(),
args
);
};
let (field, index_type) = match list.data_type() {
DataType::List(field) => (field, DataType::Int32),
DataType::LargeList(field) => (field, DataType::Int64),
DataType::FixedSizeList(field, _) => (field, DataType::Int32),
_ => return exec_err!("expected list, got {list}"),
};
// we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
// nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
// as datafusion will do that for us
let value = Field::new("value", field.data_type().clone(), field.is_nullable())
.with_metadata(field.metadata().clone());
let index = Field::new("index", index_type, false);
Ok(vec![None, Some(vec![value, index])])
}
}
```
</details>
> pub struct LambdaExpr {
/// Parameter names/types already resolved
pub param_types: Vec<DataType>,
/// Expression body, what needs to be evaluated, this thing potentially can be UDF
pub body: Arc<dyn PhysicalExpr>,
}
The current implementation is functional only with the parameters names without requiring their datatypes, and IHMO it's easier to the users to be like that, both expr_api and sql. The expr api could look like this:
```rust
array_transform(
col("my_array"),
lambda(
["current_value"], // define the parameters name
lambda_variable("current_value") * lit(2) // the lambda body
)
)
```
How to parse the clickhouse syntax I believe it's a question for the future: either check that the types match with the ones returned by `LambdaUDF::lambdas_parameters`, or cast the parameters to the specified type
> 1. replace null lists that is not empty underneath
> 3. fixed size list with a null list
> I'm conflicted on whether we should fix case 1 for them as it is costly and the user might have prior knowledge to avoid that.
Saw your review in #17220 but thought it mattered only for performance and didn't thought about fallible expressions..
Yeah.. maybe we can add a helper udf that clean the list on user demand, to be called before calling the lambda udf?
For fixed size lists, I think we can replace null sublists with the first non-null sublist of the array, if the non-null also fails then it's a user problem
> 2 .sliced list and not computing values outside the sliced data
Also saw on #17220 about this but forgot it, thanks for reminding me, will fix soon
> In every case here, we should answer:
>
> how would the user handle that?
> how could we make it easier for the user?
> How would we help them avoid forgetting handling?
I think the best we can do is to document it, add examples and helpers functions and add a lot of comments in one or few LambdaUDFs as reference implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3664171242
> This is ready for a first look @shehabgamin 🚀
@gstvg Exciting! I've started reviewing, it will take me a few days to get through it all. Thank you for thoroughly explaining all of the options btw, it's super helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2908204189
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
Now that I think about it, do we need to pass `ValueOrLambdaParameter<DataType>` to `coerce_value_types`? 🤔 since lambda variants will always get mapped to `None` and skipped the implementor will really never use them i think. Would it be possible to receive &[DataType] only like `coerce_types` from ScalarUDF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2966646570
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
Nice! Do you mind opening a draft PR so we can follow up there? Ben or I can take a look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2971375268
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Please add the following tests because you have bug in both of them from what I can see (and add similar tests for fixed size list)
```rust
fn create_i32_list(
values: impl Into<Int32Array>,
offsets: OffsetBuffer<i32>,
nulls: Option<NullBuffer>,
) -> GenericListArray<i32> {
let values = values.into();
let list_field = Arc::new(Field::new(
"item",
values.data_type().clone(),
values.is_nullable(),
));
GenericListArray::<i32>::new(Arc::clone(&list_field), offsets, Arc::new(values), nulls)
}
#[test]
fn transform_on_sliced_list_should_not_evaluate_on_unreachable_values() {
let list = create_i32_list(
vec![
// Have 0 here so if the expression is called on data that it will fail
0, 4, 100, 25, 20, 5, 2, 1, 10,
],
OffsetBuffer::<i32>::from_lengths(vec![1, 3, 4, 1]).slice(1, 3),
None,
);
let res = evaluate_array_transform(list, item -> 100 / item);
let ColumnarValue::Array(res) = res else {
panic!("expect array")
};
let actual_list = res.as_list::<i32>();
let expected_list = create_i32_list(
vec![25, 1, 4, 5, 20, 50, 100, 10],
OffsetBuffer::<i32>::from_lengths(vec![3, 4, 1]),
None,
);
assert_eq!(actual_list, &expected_list);
}
#[test]
fn transform_function_should_not_be_evaluated_on_values_underlying_null() {
let list = create_i32_list(
// 0 here for one of the values behind null, so if it will be evaluated
// it will fail due to divide by 0
vec![100, 20, 10, 0, 1, 2, 0, 1, 50],
OffsetBuffer::<i32>::from_lengths(vec![3, 4, 2]),
Some(NullBuffer::from(vec![true, false, true])),
);
let schema = Arc::new(Schema::new(vec![Field::new(
"col_0",
list.data_type().clone(),
true,
)]));
let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(list)]).unwrap();
let res = evaluate_array_transform(list, item -> 100 / item);
let res = transform_expr.evaluate(&batch).unwrap();
let ColumnarValue::Array(res) = res else {
panic!("expect array")
};
let actual_list = res.as_list::<i32>();
let expected_list = create_i32_list(
vec![1, 5, 10, 100, 2],
OffsetBuffer::<i32>::from_lengths(vec![3, 0, 2]),
Some(NullBuffer::from(vec![true, false, true])),
);
assert_eq!(actual_list, &expected_list);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3009828234
##########
datafusion/execution/src/task.rs:
##########
Review Comment:
Should we add a public accessor here? so users can pass lambda functions in `TaskContext::new(...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3082380442
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
Yes, but what I'm trying to say is that currently `HigherOrderFuncionExpr` determines if an argument is a value or a lambda by downcasting only it's top level arguments, and not children of it. If a lambda get's wrapped the downcast will fail and it will be treated as a normal value argument.
I can implement any of the options listed above, I just would like to know if you prefer one of them. I probably would implement a mix of options 1 & 2. Option 5 is easiest but any existing tree rewrite that wraps every expression would need to be modified. Those 5 are the only ones I could think of without increasing the amount work a udhof implementor has to perform and differentiating it too much from `ScalarUDFImpl`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3006017610
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
@LiaCastaneda Hmm `clean_null_values` sound good, implemented at https://github.com/apache/datafusion/pull/18921/changes/a9d0e6cbf4b7ddd0e3d70a4076193f93073af62a.
For variable sized lists we can also check if any null is non-empty before moving the data.
The cleanup is indeed the same everytime, so we can do it transparently
Maybe in the future we can add a `null_safe` method to `PhysicalExpr` and only cleanup if any expr of the lambda body is non null safe?
Yes, vectorized execution is a must, and if the cleanup proves to be too expensive maybe we can use it only when there are no nulls, and otherwise use a loop like Trino does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3004965802
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,104 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ ///
+ /// For example, `array_transform([1,2,3], v -> v+1)` would be equivalent to:
+ ///
+ /// ```text
+ /// LambdaFunction(array_transform)
+ /// ├── args[0]: Literal([1,2,3])
+ /// └── args[1]: Lambda
+ /// ├── params: ["v"]
+ /// └── body: BinaryExpr(+)
+ /// ├── LambdaVariable("v")
+ /// └── Literal(1)
+ /// ```
+ LambdaFunction(LambdaFunction),
Review Comment:
Should we call this a higher-order function instead? Calling it a lambda function is bit misleading since.
Is this going to be limited to only anonymous inline functions are will it be possible to use a scalar UDF name as the function parameter as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3085155465
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
I agree with @rluvaton, I think it's ok to assume the wrapper will report the wrapped expression as a child -- otherwise there is no way of finding the lambda 🤷♀️ (besides, children() is mandatory, which iiuc usually means that if it's not implemented correctly there is no guarantee of correctness). We can just traverse children() until we find the lambda.
Regarding @rluvaton's initial comment -- `args` already contains the lambda as the second argument, as @gstvg says:
```
args[0] = LiteralExpr([1, 2]) ← value array
args[1] = LambdaExpr(v -> v*2) ← lambda
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3022045503
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
It's either this, `DataType::Null` or an error. I don't have a strong opinion on this, actually. Right now, the only place I call it is literally like this:
```rust
.map(|arg: Expr| {
let field = arg.to_field(schema)?.1;
match arg {
Expr::Lambda(_lambda) => Ok(ValueOrLambda::Lambda(field)),
_ => Ok(ValueOrLambda::Value(field)),
}
})
```
We can just change it to
```rust
.map(|arg: Expr| {
match arg {
Expr::Lambda(lambda) => Ok(ValueOrLambda::Lambda(lambda.body.to_field(schema)?.1)),
_ => Ok(ValueOrLambda::Value(arg.to_field(schema)?.1)),
}
})
```
During higher order function type coercion, arguments are ignored if they are a lambda and only value arguments are checked and coerced so the return type for a lamba is irrelevant.
Some other expressions return `DataType::Null` like logical `Placeholder` without a specifed type, `Wildcard` and `GroupingSet` and physical `NoOp` and `UnknownColumn`. If you think that's better I'm happy to change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074567921
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
+ index: usize,
+ field: FieldRef,
+}
+
+impl Eq for LambdaVariable {}
+
+impl PartialEq for LambdaVariable {
+ fn eq(&self, other: &Self) -> bool {
+ self.name == other.name && self.field == other.field
+ }
+}
+
+impl Hash for LambdaVariable {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name.hash(state);
+ self.field.hash(state);
+ }
+}
+
+impl LambdaVariable {
+ /// Create a new lambda variable expression
+ pub fn new(name: String, index: usize, field: FieldRef) -> Self {
+ Self { name, index, field }
+ }
+
+ /// Get the variable's name
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Get the variable's index
+ pub fn index(&self) -> usize {
+ self.index
+ }
+
+ /// Get the variable's field
+ pub fn field(&self) -> &FieldRef {
+ &self.field
+ }
+}
+
+impl std::fmt::Display for LambdaVariable {
+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+ write!(f, "{}@", self.name)
+ }
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ if self.index >= batch.num_columns() {
+ return internal_err!(
+ "PhysicalExpr LambdaVariable references column '{}' at index {} (zero-based) but batch only has {} columns: {:?}",
+ self.name,
+ self.index,
+ batch.num_columns(),
+ batch
+ .schema_ref()
+ .fields()
+ .iter()
+ .map(|f| f.name())
+ .collect::<Vec<_>>()
+ );
+ }
+
+ Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ vec![]
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ _children: Vec<Arc<dyn PhysicalExpr>>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ Ok(self)
+ }
+
+ fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{}", self.name)
+ }
+}
+
+/// Create a lambda variable expression
+pub fn lambda_variable(
+ name: impl Into<String>,
+ field: FieldRef,
+ schema: &Schema,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let name = name.into();
+ let index = schema.index_of(&name)?;
+
+ let schema_field = schema.field(index);
+
+ if field.as_ref() != schema_field {
+ return plan_err!(
+ "LambdaVariable owned field differ from schema field {field} != {schema_field}"
+ );
+ }
Review Comment:
included at https://github.com/apache/datafusion/pull/18921/changes/2259f70e727d61b8e0e02f01556635fedf694891
I implemented this too tailored to the call at `create_physical_expr`, where I think we should check if the fields match, but it was indeed bad, thanks. I removed the field arg but also inlined the old implementation at `create_physical_expr`
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
+ index: usize,
+ field: FieldRef,
+}
+
+impl Eq for LambdaVariable {}
+
+impl PartialEq for LambdaVariable {
+ fn eq(&self, other: &Self) -> bool {
+ self.name == other.name && self.field == other.field
+ }
+}
+
+impl Hash for LambdaVariable {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name.hash(state);
+ self.field.hash(state);
+ }
+}
+
+impl LambdaVariable {
+ /// Create a new lambda variable expression
+ pub fn new(name: String, index: usize, field: FieldRef) -> Self {
+ Self { name, index, field }
+ }
+
+ /// Get the variable's name
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Get the variable's index
+ pub fn index(&self) -> usize {
+ self.index
+ }
+
+ /// Get the variable's field
+ pub fn field(&self) -> &FieldRef {
+ &self.field
+ }
+}
+
+impl std::fmt::Display for LambdaVariable {
+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+ write!(f, "{}@", self.name)
+ }
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ if self.index >= batch.num_columns() {
+ return internal_err!(
+ "PhysicalExpr LambdaVariable references column '{}' at index {} (zero-based) but batch only has {} columns: {:?}",
+ self.name,
+ self.index,
+ batch.num_columns(),
+ batch
+ .schema_ref()
+ .fields()
+ .iter()
+ .map(|f| f.name())
+ .collect::<Vec<_>>()
+ );
+ }
+
+ Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
Review Comment:
included at https://github.com/apache/datafusion/pull/18921/changes/2259f70e727d61b8e0e02f01556635fedf694891 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3017194842
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
Is it an issue that this is technically incorrect? The type of the lambda expression itself is actually a function type, where here it's presented as just its return type. Can this lead to incorrect type analysis?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton closed pull request #18921: Add lambda support and array_transform udf
URL: https://github.com/apache/datafusion/pull/18921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4340706710
Many thanks for everyone involved, both reviewers and also those who showed interest on the feature. Reviewing such a big PR is not easy and I'm very grateful for it, thanks again :heart:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3684901556
@linhr Thanks, I got it now. I think it's possible to resolve lambda parameters and remove them from the arguments list with `ScalarUDF` itself, with a few cons compared to a dedicated to `LambdaUDF`, but with much less code changes and similar enough to compare the alternatives. I believe I'm already finishing it, and will push to another branch soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3004938118
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
fwiw I looked into how Trino handles this, and they simply don't apply the function in a vectorized way -- they have [one for loop](https://github.com/trinodb/trino/blob/2aae427ebf8e18477cec163065df4ae0f4575307/core/trino-main/src/main/java/io/trino/operator/scalar/ArrayTransformFunction.java#L199) that checks for nulls and applies the lambda per element. IMO it's great to be able to execute this in a vectorized way in DataFusion even if the price to pay is the null cleanup (which is only necessary on arrays with nulls).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2902630414
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,334 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScalarUDFImpl`] definitions for array_transform function.
+
+use arrow::{
+ array::{
+ Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray,
+ RecordBatch, RecordBatchOptions,
+ },
+ compute::take_record_batch,
+ datatypes::{DataType, Field, FieldRef, Schema},
+};
+use datafusion_common::{
+ exec_err,
+ tree_node::{
+ Transformed, TransformedResult, TreeNode, TreeNodeRecursion, TreeNodeRewriter,
+ },
+ utils::{elements_indices, list_indices, list_values, take_function_args},
+ HashMap, Result,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaUDF, Signature,
+ ValueOrLambda, ValueOrLambdaField, ValueOrLambdaParameter, Volatility,
+};
+use datafusion_macros::user_doc;
+use datafusion_physical_expr::expressions::{LambdaExpr, LambdaVariable};
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::{any::Any, sync::Arc};
+
+//make_udf_expr_and_func!(
+// ArrayTransform,
+// array_transform,
+// array lambda,
+// "transforms the values of a array",
+// array_transform_udf
+//);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: Signature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
Yes, thanks https://github.com/apache/datafusion/pull/18921/commits/34137e15ca87bd71a9a0b2bd386a823f81fafad8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4236543783
Hi @rluvaton @comphead @alamb -- I think this PR is in good shape (cc @gstvg, please correct me if I missed anything or if there are any blockers I’m not aware of). Some of us have already taken a look. There are still some follow-ups tracked in this epic: https://github.com/apache/datafusion/issues/21172, and @gstvg has draft PRs for several of them. Since this is implemented as a separate trait and doesn’t touch existing paths, it seems relatively safe to merge. Would anyone have time to take another look or share thoughts on how we should proceed? I believe there’s quite a bit of interest in this feature, so it would be great to get some guidance on how to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073470703
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
Review Comment:
```suggestion
"UDHOF {} returned a different number of rows than expected. Expected: {}, Got: {}",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2982897288
##########
datafusion/physical-expr/src/expressions/lambda.rs:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda expression: [`LambdaExpr`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+use datafusion_common::plan_err;
+use datafusion_common::{HashSet, Result, internal_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents a lambda with the given parameters names and body
+#[derive(Debug, Eq, Clone)]
+pub struct LambdaExpr {
+ params: Vec<String>,
+ body: Arc<dyn PhysicalExpr>,
+}
Review Comment:
Also, when capture support is added again, `LambdaArgument::evaluate` will have a new argument to adjust the captured columns relative to the lambda parameters.
```rust
fn evaluate(
&self,
args: &[&dyn Fn() -> Result<ArrayRef>],
mut adjust: impl FnMut(&[ArrayRef]) -> Result<Vec<ArrayRef>>,
) -> Result<ColumnarValue>
```
https://github.com/apache/datafusion/pull/18921/changes/6f2c92bd0858aa57cf5034a830a8df23829ac9b1#diff-454a60ee653df3b3601c9bc26b2cfd48a92b78d561b2a271b140466786d0ff51R208-R211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3953563408
I had a partial implementation for `array_transform` in https://github.com/apache/datafusion/pull/17289 but I wasn't able to get it over the line before I had to work on other things. There is a *lot* of discussion here so I'm going to try to make some time in the next few days to go over it all, but I am very interested in the directions this goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3944235156
thanks a lot for this PR.
Couple of things to make sure we support or have a way to add them in the future without breaking changes:
1. Support `Map`, `(Large)List`, `(Large)ListView`, `FixedSizeList` as the input for lambda
2. multiple lambdas in a single expression, for example `map_key_value(some_map_col, map_key_lambda, map_value_lambda)` and each lambda gets a different variables
3. Lambda expression that access columns that are not in the list itself
so I can do the following:
```
| year | grades |
|------|----------------|
| 1998 | [1, 2, 3] |
| 1999 | [4, 99, 5, 10] |
| 2000 | [6, 0, null] |
```
`array_transform(grades, x -> if year <= 1990 then x * 10 else x)`
4. optional arguments for lambda, for example the index of the item in the list
the optional here is important as I want to avoid creating that input if I don't need to.
5. Nested lambda expressions: `array_transform(matrix, x -> array_transform(x, y -> y * 2))`
And some stuff that every lambda expression would need that we would need to provide a helpers (and not fix the input IMO as it would be expensive and the user might be able to have some prior knowledge on the input or just want their own implementation, or the child lambda expression can't error)
> (I have helpers for all of these)
1. how we handle null lists when the underlying list is not empty and the expression can fail, for example: `array_transform(list, x -> 1 / x)`
for example this input: which the second list is `null` but the underlying value is `[0, 3]` which if we run the transform on it it will fail with division by zero.
```rust
fn get_list() -> GenericListArray<i32> {
GenericListArray::new(
Arc::new(Field::new_list_field(DataType::Int8, false)),
OffsetBuffer::<i32>::from_lengths(vec![2, 2, 1]),
Arc::new(Int8Array::from(vec![1, 2, 0, 3, 4])),
Some(NullBuffer::from(&[true, false, true])),
)
}
```
I have a lot of helpers to cleanup the nulls BTW
2. How we handle sliced lists the child should only work on the sliced data.
-----
I think having a new `LambdaUDFImpl` is better than adding functions on existing `ScalarUDF` because:
1. The `ScalarUDF` trait will not grow too much and make implementing regular scalar UDFs easier or lambda overwhelming
2. what if we need to add a required function but only for lambda, we can add it on the new trait with ease and we won't need to do some weird stuff
to avoid breaking changes.
3. Less ambiguity on the API.
----
I want to keep the simplicity of `ScalarUDF` which means that in order to evaluate a lambda expression I don't need to construct stuff, only need to provide the input and maybe some options for future use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3954404061
You are right @comphead, my first implementation was more complex and I thoroughly documented it, but after simplifying it to the current version, I thought it was easy to grasp and poorly documented it, but I was obviously wrong. Despite not liking much my high level writing skills, I believe that writing the doc is my responsibility (there's few edge cases and alternative approaches not discussed yet). But I would really appreciate if you reviewed it, I can push a `DOC.md` to this branch and we can start a review on it to not add even more comments here, and then finally update the PR description, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2982657524
##########
datafusion/common/src/utils/mod.rs:
##########
@@ -971,11 +971,27 @@ pub fn take_function_args<const N: usize, T>(
})
}
+/// Returns the inner values of a list, or an error otherwise
+/// For [`ListArray`] and [`LargeListArray`], if it's sliced, it returns a
+/// sliced array too. Therefore, too reconstruct a list using it,
+/// you must adjust the offsets using [`adjust_offsets_for_slice`]
Review Comment:
Also fixed at https://github.com/apache/datafusion/commit/6ae73cb4a569d6693d37b601cca38a4c689722df
Since this was just to efficiently handle sliced lists, and not to correctness, it was scraped at https://github.com/apache/datafusion/pull/18921/changes/6f2c92bd0858aa57cf5034a830a8df23829ac9b1 to make the PR smaller but I forgot to update the comment, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3002224991
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
🤔 This is a good point I was not aware of. It's tricky because in `array_transform` this could be a problem or not depending on which expression it gets (if it's a multiplication it's not an issue but if it's a division it is), so I imagine the best option here is to be conservative and always clean up.
Although, we do have cheap ways to know if arrow Arrays have nulls (`array.null_count()` etc), so we could skip the cleanup entirely when there are no nulls. I guess this cleanup would be the same for all lambda functions, so it probably makes sense to handle it transparently in datafusion before calling the user's invoke That way implementors never have to think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "benbellick (via GitHub)" <gi...@apache.org>.
benbellick commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2896415393
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
I will address this comment more fully later. The protos for lambdas do exist in `substrait-rs` but haven't been released yet. I am looking into how to do that release right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "benbellick (via GitHub)" <gi...@apache.org>.
benbellick commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2920166967
##########
datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:
##########
Review Comment:
I have replace that PR with a new one due to the broken `substrait-rs` release. It has now been fixed and the correct version is `v0.63.0`
PR: https://github.com/apache/datafusion/pull/20876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2996310278
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ let (list, _lambda) = value_lambda_pair(self.name(), args)?;
+
+ let field = match list.data_type() {
+ DataType::List(field) => field,
+ DataType::LargeList(field) => field,
+ DataType::FixedSizeList(field, _) => field,
Review Comment:
ah I completely missed the fact that this was already handled by type coercion above, you are right, for a first version it's probably fine as it is now. We can implement the map_transform after as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3004965802
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,104 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ ///
+ /// For example, `array_transform([1,2,3], v -> v+1)` would be equivalent to:
+ ///
+ /// ```text
+ /// LambdaFunction(array_transform)
+ /// ├── args[0]: Literal([1,2,3])
+ /// └── args[1]: Lambda
+ /// ├── params: ["v"]
+ /// └── body: BinaryExpr(+)
+ /// ├── LambdaVariable("v")
+ /// └── Literal(1)
+ /// ```
+ LambdaFunction(LambdaFunction),
Review Comment:
Should we call this a higher-order function instead? Calling it a 'lambda function' is bit misleading. In the literature at least that's the name used for the `Lambda` type below.
Is this going to be limited to only anonymous inline functions are will it be possible to use a scalar UDF name as the function parameter as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3004965802
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,104 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ ///
+ /// For example, `array_transform([1,2,3], v -> v+1)` would be equivalent to:
+ ///
+ /// ```text
+ /// LambdaFunction(array_transform)
+ /// ├── args[0]: Literal([1,2,3])
+ /// └── args[1]: Lambda
+ /// ├── params: ["v"]
+ /// └── body: BinaryExpr(+)
+ /// ├── LambdaVariable("v")
+ /// └── Literal(1)
+ /// ```
+ LambdaFunction(LambdaFunction),
Review Comment:
Should we call this a higher-order function instead? Calling it a 'lambda function' is bit misleading. In the literature at least that's the name used for the `Lambda` type below.
Is this going to be limited to only anonymous inline functions or will it be possible to pass a scalar UDF name as the function parameter as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3006009469
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
I just run into an issue when upgrading to DataFusion 52 internally, and this will have the same problem.
`CASE WHEN` expression does column projection as a way of optimization to avoid copy unused column, it was added by @pepijnve in:
- #18329
the way it does that is transforming the tree and finding Column, so if you have `array_transform(list, |v| case when v > 1 then 2 when v > 4 then 6 else 8)` it will use the path with column projection and it will fail to mark this as accessing a column and thus, will not work.
make sure the CASE WHEN is similar to my example so it will use that path.
column you please add 3 tests:
1. `array_transform` with that case when only on lambda variable
2. `array_transform` with case when on lambda variable and external column
3. `array_transform` with only external columns.
I've opened a bug, but I just want to make sure we have tests for this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3640819962
Thanks a lot @gstvg, huge amount of work apparently (even if I am not enough skilled to judge). I am waiting for such functions since a very long time.. IMHO, this is a MUST HAVE in datafusion. All serious analytics engine provide this kind of functions, DuckDB implementation is very mature, Polars seems to have it as well even if I didn"t tried it yet and I really hope that datafusion will jump into this enhancement very quickly as it is again a big miss.
Use cases like list_filter(list, expr), list_map(list, expr) and list_reduce(list, base, expr) will be available soon.
Again, thanks a lot for your work and I cross the fingers for a quick progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3953471440
Thanks @gstvg I feel we need some structure here.
I pinged in ASF slack that we are discussing lambda support in this RFC. Also it would be great for ppl not to read through comments but having some simple doc on proposed object and how it would be parsed/evaluated, what traits involved.
I can try to start this doc by going through this RFC, or if you feel more comfortable you can create it, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2985097129
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
Review Comment:
Hmm actually yes, haven't thought about this before https://github.com/apache/datafusion/pull/18921/changes/39db62ba532d7a1349c1c275c14b8db71a291b6d
I kept `value_fields_with_lambda_udf` working with the `ValueOrLambda` enum because it needs to check the number of all arguments for `LambdaTypeSignature::Any(usize)` and if/when we add `TypeSignature::Exact` equivalent it will likely define the exact position of lambdas, which will require full positional info
Just one minor thing to note is that both changes at `coerce_value_types` and `lambas_parameters` postpone the detection of wrong parameters positions to `return_field_from_args`, which it's not ideal but it's not a big deal either
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3655460423
Hello @gstvg, thanks for your last submission, some questions came to my mind reading your proposal, I see quite well the map/filter operations that are producing array in result but I am a bit "confused" about the support of "reduce" like operation that could produce any type as output like another array or scalar values with an aggegation value that has to be first intialized by an Expr through the call to HoF and the "propagation" of this value accross the plan architecture of datafusion.
I think about use cases like **array_reduce(input: [], init_aggregate_val: Expr, lamba: Expr)** with **lambda(current_value_of_input, aggregate_value) -> aggregate_value**
Is it possible with your proposal ? (I saw potential several parameters for the lambda so I suppose your proposal is compliant ?)
Also, is it possible to "nest" the HoF like: array_transform(arr_transform([], lamda1), lamba2) ?
For this last question, I don't see any "obstacles" but I prefer to confirm with you... :-)
And also, pratically, how to reference "current_value" and "aggregate_value" in the dataframe API? These are not named in the schema and these are not query parameter as well so it's not clear for me how to reference these values inside the expressions:
**df.array_transform(col("my_array"), max(<????> * 2, 1000)**
Sorry to question like that, but I was bumping my head on the walls on this topic since months without success (not enough mastering datafusion logical/physical plans logic).
Many, many thanks !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3667071456
Crystal clear @gstvg thanks a lot, cannot wait to test and use it.
A bit disappointed for reduce operation but i fully understand the drawbacks for the reviewers.
Hope this review will be successfull.
And thanks in advance to all reviewers and of course to the author of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3081130002
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ if self.return_field.is_nullable() != nullable {
+ Arc::make_mut(&mut self.return_field).set_nullable(nullable);
+ }
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
Review Comment:
@rluvaton
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4244980355
@rluvaton @LiaCastaneda merge conflicts and the comments have been resolved (except for https://github.com/apache/datafusion/pull/18921#pullrequestreview-4099223423).
Since the last review, I also pushed 2 minor commits, one to include the index in lambda variable formatting https://github.com/apache/datafusion/pull/18921/commits/5c2c72a93331c55cb45fa45335e3e994840fb251 and one to include higher order functions with scalar functions at the sql user guide https://github.com/apache/datafusion/pull/18921/commits/3c9fe3975939f52e751a5ad00aa0db2c178d6ad1 (I missed that and they weren't being rendered anywhere)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073422413
##########
datafusion/expr/src/type_coercion/functions.rs:
##########
@@ -148,6 +151,81 @@ pub fn fields_with_udf<F: UDFCoercionExt>(
.collect())
}
+/// Performs type coercion for higher order function arguments.
+///
+/// For value arguments, returns the field to which each
+/// argument must be coerced to match `signature`.
+/// For lambda arguments, returns a clone of the associated data
+///
+/// For more details on coercion in general, please see the
+/// [`type_coercion`](crate::type_coercion) module.
+pub fn value_fields_with_higher_order_udf<L: Clone>(
Review Comment:
can you please add tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073479590
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
it should include the function as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073442031
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
+ index: usize,
+ field: FieldRef,
+}
+
+impl Eq for LambdaVariable {}
+
+impl PartialEq for LambdaVariable {
+ fn eq(&self, other: &Self) -> bool {
+ self.name == other.name && self.field == other.field
+ }
+}
+
+impl Hash for LambdaVariable {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name.hash(state);
+ self.field.hash(state);
+ }
+}
+
+impl LambdaVariable {
+ /// Create a new lambda variable expression
+ pub fn new(name: String, index: usize, field: FieldRef) -> Self {
+ Self { name, index, field }
+ }
+
+ /// Get the variable's name
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Get the variable's index
+ pub fn index(&self) -> usize {
+ self.index
+ }
+
+ /// Get the variable's field
+ pub fn field(&self) -> &FieldRef {
+ &self.field
+ }
+}
+
+impl std::fmt::Display for LambdaVariable {
+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+ write!(f, "{}@", self.name)
+ }
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ if self.index >= batch.num_columns() {
+ return internal_err!(
+ "PhysicalExpr LambdaVariable references column '{}' at index {} (zero-based) but batch only has {} columns: {:?}",
+ self.name,
+ self.index,
+ batch.num_columns(),
+ batch
+ .schema_ref()
+ .fields()
+ .iter()
+ .map(|f| f.name())
+ .collect::<Vec<_>>()
+ );
+ }
+
+ Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
Review Comment:
can you verify that the type here actually match the field?, since I can pass whatever field I want and it will pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3674167149
Thanks @linhr
I think a new `LambdaUDF` is reasonable, and I'm not strongly inclined towards using `ScalarUDF` either
>Expr::Lambda and Expr::LambdaColumn are only fragments whose data types etc. are not well-defined which may be challenging to work with in other parts of the library (e.g. ExprSchemable).
Yeah, this is the major challenge in this PR. Currently, `Expr::Lambda` always return `DataType::Null` and `Expr::LambdaVariable` embeds a `FieldRef` which is used to implement the `ExprSchemable` and don't even look at the `DFSchema`
>when the function is actually "invoked" with Arrow arrays, the lambda parameter should have been resolved and removed from the argument list.
Could you expand this further? It is like `LambdaUDF` having a `resolve_lambdas` method which result is passed to the `invoke_with_args` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "keen85 (via GitHub)" <gi...@apache.org>.
keen85 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3796510353
@gstvg any updates on this one? 😇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4333878013
@gstvg I'm closing this branch since the new pr was merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2902643645
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
> I imagine type coercion for lambda functions would only apply to value arguments right?
Yes https://github.com/apache/datafusion/pull/18921/commits/3ded1154a4aa7adc167ec033622308c2baa8d524
Honestly this should have been done when copying from `ScalarUDFImpl` but I missed it, thanks for catching it. I also added a `LambdaTypeSignature` with a minimal set of variants so that `coerce_value_types` can configured to be used or not, just like in `ScalarUDF`, and included it on `LambdaSignature`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2895902831
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
@benbellick added some in the substrait spec in https://github.com/substrait-io/substrait/pull/889, but might not be released yet. I wonder if the mappig can be done 1-1
##########
datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:
##########
Review Comment:
should we also add an unimplemented note in the [substrait_consumer](https://github.com/apache/datafusion/blob/678d1ad7f4590e74e7bae0326292949617da0f57/datafusion/substrait/src/logical_plan/consumer/expr/mod.rs#L54)?
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
I imagine type coercion for lambda functions would only apply to value arguments right?
For example, arr in `array_transform(arr, v -> v)` if arr is `List<Int32>` but the function only handles `List<Int64>`, people might forget to coerce this themselves.
What if we changed the `coerce_types` signature of `LambdaUDF` to:
`fn coerce_types(&self, args: &[ValueOrLambdaParameter]) -> Result<Vec<Option<DataType>>>`
So it coerces value args (returning Some(DataType)) but skips lambda args (returning None)? I think `coerce_value_types` would be a better name for this
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,334 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScalarUDFImpl`] definitions for array_transform function.
+
+use arrow::{
+ array::{
+ Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray,
+ RecordBatch, RecordBatchOptions,
+ },
+ compute::take_record_batch,
+ datatypes::{DataType, Field, FieldRef, Schema},
+};
+use datafusion_common::{
+ exec_err,
+ tree_node::{
+ Transformed, TransformedResult, TreeNode, TreeNodeRecursion, TreeNodeRewriter,
+ },
+ utils::{elements_indices, list_indices, list_values, take_function_args},
+ HashMap, Result,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaUDF, Signature,
+ ValueOrLambda, ValueOrLambdaField, ValueOrLambdaParameter, Volatility,
+};
+use datafusion_macros::user_doc;
+use datafusion_physical_expr::expressions::{LambdaExpr, LambdaVariable};
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::{any::Any, sync::Arc};
+
+//make_udf_expr_and_func!(
+// ArrayTransform,
+// array_transform,
+// array lambda,
+// "transforms the values of a array",
+// array_transform_udf
+//);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: Signature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
would it be clearer if we had a especial signature for lambda functions that has both volatility and parameter names?
```
pub struct LambdaSignature {
pub volatility: Volatility,
pub parameter_names: Option<Vec<String>>,
}
```
since this is a different trait to ScalarUDF, and types can't really de defined in an static way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4354272485
@gstvg thanks a lot for your work, cannot wait for future release and test it ! Thanks also to all reviewers. 🥰
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2971361327
##########
datafusion/common/src/utils/mod.rs:
##########
@@ -971,11 +971,27 @@ pub fn take_function_args<const N: usize, T>(
})
}
+/// Returns the inner values of a list, or an error otherwise
+/// For [`ListArray`] and [`LargeListArray`], if it's sliced, it returns a
+/// sliced array too. Therefore, too reconstruct a list using it,
+/// you must adjust the offsets using [`adjust_offsets_for_slice`]
Review Comment:
You are not returning the sliced values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2971383532
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
I don't think the lambda variable should be determined by the name, and instead it should be by the index like Column. no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3022045503
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
It's either this, `DataType::Null` or an error. I don't have a strong opinion on this, actually. Right now, the only place I call it is literally like this:
```rust
.map(|arg: Expr| {
let field = arg.to_field(schema)?.1;
match arg {
Expr::Lambda(_lambda) => Ok(ValueOrLambda::Lambda(field)),
_ => Ok(ValueOrLambda::Value(field)),
}
})
```
We can just change it to
```rust
.map(|arg: Expr| {
match arg {
Expr::Lambda(lambda) => Ok(ValueOrLambda::Lambda(lambda.body.to_field(schema)?.1)),
_ => Ok(ValueOrLambda::Value(arg.to_field(schema)?.1)),
}
})
```
During higher order function type coercion, lambda arguments are ignored and only value arguments are checked and coerced so the return type for a lamba is irrelevant.
Some other expressions return `DataType::Null` like logical `Placeholder` without a specifed type, `Wildcard` and `GroupingSet` and physical `NoOp` and `UnknownColumn`. If you think that's better I'm happy to change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976056024
##########
datafusion/physical-expr/src/expressions/lambda.rs:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda expression: [`LambdaExpr`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+use datafusion_common::plan_err;
+use datafusion_common::{HashSet, Result, internal_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents a lambda with the given parameters names and body
+#[derive(Debug, Eq, Clone)]
+pub struct LambdaExpr {
+ params: Vec<String>,
+ body: Arc<dyn PhysicalExpr>,
+}
Review Comment:
`LambdaArgument::evaluate` receives closures instead of eager evaluated arguments as a batch and it allows skiping the evaluation of unnecessary parameters. I also like that it encapsulates the internal lambda implementation allowing for less breaking changes in the future (I changed it a few times and I wouldn't be surprised if it must be changed again)
When we add capture support, `LambdaArgument` will receive a `captures: Option<RecordBatch>` field, which will differentiate it even more from `LambdaExpr`
`LambdaArgurment.params` being `FieldRef`' are used to construct the evaluation batch with precise nullability, otherwise we could only use the param datatype and set nullable to always false.
Making `LambdaExpr.params` `FieldRef` instead of strings would complicate `expr_api` users as they would need to provide the Field instead of only the name
And since `LambdaExpr` should only exist as a lambda function argument, where it's specially handled, I think it's good to error with a clear message if it ends up outside a lambda function , instead of likely failing with a missing field or mismatched types error, or worse, not failing at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976104051
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
To make this PR smaller and easier to reason about, I used the name for now. Could have added a comment, though.
Is possible to resolve by index, we just need to decide how to it supporting the following `expr_api` usage:
```rust
// v -> v + v
lambda(
["v"],
lambda_var("v") + col("v")
)
```
Resolving by name as today, the code above works without names conflicts. So that it keeps working by index we can either:
1. During lambda function physical planning, we create a new schema for each lambda argument that includes it's parameters and use it to plan the lambda body. For each lambda parameter on that schema, add a metadata entry to them so that when theirs variables are being planned, we search for the index of field with both the variable name *and* the metadata entry, but it feels a bit loose?
2. Use different schema for columns and parameters, and move it around either via a new argument on `create_physical_expr` (breaking) or embed it on `ExecutionProps`, which is still breaking but less, but also doesn't feel like a good fit for `ExecutionProps`
3. Revert to the old approach where `LambdaVariable` holds a `Option<ArrayRef>` that should be binded for each evaluation, and won't depend on the planning schema nor the evaluation batch
4. Resolve by the index but don't compute them during the regular physical planning, via `create_physical_expr`, but right after it with a tree node rewriter that computes the variables indexes without depending on the planning schema.
Note that this does not apply to sql planning as there lambda variables have higher precedence and columns get shadowed by them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976010365
##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -744,6 +746,38 @@ impl TreeNodeRewriter for TypeCoercionRewriter<'_> {
});
Ok(Transformed::yes(new_expr))
}
+ Expr::LambdaFunction(LambdaFunction { func, args }) => {
+ let current_fields = args
+ .iter()
+ .map(|arg| match arg {
+ Expr::Lambda(_) => Ok(ValueOrLambda::Lambda(())),
+ _ => Ok(ValueOrLambda::Value(arg.to_field(self.schema)?.1)),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let new_fields =
+ value_fields_with_lambda_udf(¤t_fields, func.as_ref())?;
+
+ let transformed = current_fields != new_fields;
+
+ let new_args = if transformed {
Review Comment:
https://github.com/apache/datafusion/pull/18921/changes/547c1488b37a929c55ed6adeb625a6fba35e341c
It was just a micro-optimization, but now is easier to read, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2975999590
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,109 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ LambdaFunction(LambdaFunction),
+ /// A Lambda expression with a set of parameters names and a body
+ Lambda(Lambda),
+ /// A named reference to a lambda parameter
+ LambdaVariable(LambdaVariable),
Review Comment:
https://github.com/apache/datafusion/pull/18921/changes/d75dfe3cad6560a1b105dbf013b33e19d72a469b
Thanks for the diagram, I just added `params: ["v"]` to lambda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3017194842
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
Is it an issue that this is technically incorrect? The type of the lambda expression itself is actually a function type, where here it's presented as just its return type. Can this lead to incorrect type analysis?
I haven't thought this through fully, but would it maybe be better to not have lambda functions be an `Expr`? Lambda functions don't really fit into the arrow type system. You would then have ot use an enum type that's either `Expr` or `Lambda` for the higher-order function parameters. Maybe that greatly complicates everything else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3006009469
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
I just run into an issue when upgrading to DataFusion 52 internally, and this will have the same problem.
`CASE WHEN` expression does column projection as a way of optimization to avoid copy unused column, it was added by @pepijnve in:
- #18329
the way it does that is transforming the tree and finding Column, so if you have `array_transform(list, |v| case when v > 1 then 2 when v > 4 then 6 else 8)` it will use the path with column projection and it will fail to mark this as accessing a column and thus, will not work.
make sure the CASE WHEN is similar to my example so it will use that path.
column you please add 3 tests:
1. `array_transform` with that case when only on lambda variable
2. `array_transform` with case when on lambda variable and external column
3. `array_transform` with only external columns.
I've opened a bug, but I just want to make sure we have tests for this:
- #21231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3002322975
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Maybe a middle ground solution would be introducing a trait method with a default value so implementors can indicate whether they want DataFusion to handle this or not (or the lambda function is safe or not):
```
fn clean_null_values(&self) -> bool {
true
}
```
edit: maybe it makes more sense to provide a function to indicate if the lambda is null safe or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3007849426
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
Review Comment:
https://github.com/apache/datafusion/pull/18921/changes/1b7f4bfd2c3aa9d9e3c431775e199dc4113e27d3 thanks for the report
Also added a test with case with an inner array_transform where the optimization should not try to project the lambda variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073388949
##########
datafusion/common/src/utils/mod.rs:
##########
Review Comment:
I will try to create PR to arrow with my utils to handle nulls and list sliced
##########
datafusion/common/src/utils/mod.rs:
##########
Review Comment:
I will try to create PR to arrow with my utils to handle nulls and list sliced but don't wait for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3075128457
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
Based on what you said on https://github.com/apache/datafusion/issues/21231#issuecomment-4188619900 , if the `LambdaExpr` get's wrapped with a debug expression that print the input and output, or add timing, then lambda evaluation will break at http://github.com/apache/datafusion/pull/18921/changes#diff-82f03654507864698a026045144642b3dcda8aedc20d29aea782f28b47b9d0eaR245
What I imagine could fix:
1. Since those wrapper expressions should likely return only the wrapped expression as children(in this case the lambda), we could repeatedly call `expr.children()` until we find a `LambdaExpr`, as long as every call return a single children, otherwise return an error. And then wrap the lambda *body* with those wrappers to make sure they actually get called
2. Get lambdas positions on `HigherOrderFunction::try_new` (but not on `HigherOrderFunction::new` as it called during tree rewrites and a wrapped lambda during one would be missed). Then we can check if an arg is a lambda or not by it's position and not via downcasting, and evaluate the lambda as a regular expr via it's `PhysicalExpr::evaluate` somehow
3. Add `as_inner_any` method to `PhysicalExpr` which by default returns `as_any` but that should return `as_inner_any` for "wrapper" expressions
4. Document that "wrapper" expressions `as_any` should return the wrapped expressionns instead of itself
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2896741915
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
nice, I agree the substrait consuming /producing can be left as a follow up, should we bump substrait in datafusion in the meantime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2898017709
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
Thanks @LiaCastaneda for the info and @benbellick for the release
> I wonder if the expressions can actually be mapped 1:1
I did a quick check and I believe it's possible, yes.
`LambdaFunction` produces a regular substrait `ScalarFunction`, and to consume we just check whether any `LambdaUDF` exists with the given function name and/or if any arg is a lambda.
To produce a substrait [Lambda](https://substrait.io/expressions/lambda_expressions/#lambda-expression-structure) with the field `parameters` struct where each field corresponds to a parameter, we use the return of `LambdaUDF::lambdas_parameters`, and for consuming, we can use default parameters names like `p1`, `p2` etc
To produce a [FieldReference](https://substrait.io/expressions/field_references/)/[LambdaParameterReference](https://substrait.io/expressions/lambda_expressions/#parameter-references) with a correct `steps_out` and `direct_reference.struct_field.field`, we keep a `HashMap<String, (usize, usize)>` of `lambda_parameter_name => (step_outs, struct_field_index)`, that get's updated for every `Lambda` to be produced: `step_outs` is reset to 0 for shadowed parameters, others incremented by one, and `struct_field_index` set according to it's position within the parameters of the lambda it originates from. Consuming into `LambdaVariable` with it's field resolved is similar to SQL planning: keep a `HashMap<usize, FieldRef>` of `direct_reference.struct_field.field => field` that get's updated for every `LambdaUDF` by the return of it's `lambdas_parameters` method
I'll address your another comments and then implement substrait support on another branch to make sure is viable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2906112849
##########
datafusion/substrait/src/logical_plan/producer/expr/mod.rs:
##########
@@ -152,6 +152,9 @@ pub fn to_substrait_rex(
not_impl_err!("Cannot convert {expr:?} to Substrait")
}
Expr::Unnest(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"),
+ Expr::LambdaFunction(expr) => producer.handle_lambda_function(expr, schema),
+ Expr::Lambda(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
+ Expr::LambdaVariable(expr) => not_impl_err!("Cannot convert {expr:?} to Substrait"), // not yet implemented in substrait-rs
Review Comment:
> we can use default parameters names like p1, p2 etc
Yeah @benbellick and were discussing this the other day and thought of the same workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2902626011
##########
datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:
##########
Review Comment:
Yes, thanks https://github.com/apache/datafusion/pull/18921/commits/83dfbdd13e98be8a00f90becc26f5b9f277eebff
I believe we are going to produce a regular `ScalarFunction` for `LambdaFunction` nodes, so I added the note where scalar functions are handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2924752901
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
I guess the contract of coerce_value_types would be
"return coerced types for value args in the same order you received them." but its the same assumption from `coerce_types` from ScalarUDF right?
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
I guess the contract of `coerce_value_types` would be
"return coerced types for value args in the same order you received them." but its the same assumption from `coerce_types` from ScalarUDF right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "benbellick (via GitHub)" <gi...@apache.org>.
benbellick commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4057974024
👋 Hello from the @substrait-io side of things!
I am excited to get this one in! As part of trying to help, I'm working on getting a branch off of this one working with substrait. To bring in lambda suppport, I had to merge current main (which has lambda-containing `substrait-rs` `v0.63.0`), but there were a bunch of merge conflicts. I tried to resolve them the best that I (and claude) could, but if this branch is able to either rebase off of main or merge main into it, that would help a lot! Of course, I can also just update that dependency in my local branch as well.
I also may have some questions that come up as I am implementing things. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2628576483
##########
datafusion/expr/src/expr.rs:
##########
@@ -398,6 +398,30 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Lambda expression
+ Lambda(Lambda),
+ LambdaColumn(LambdaColumn),
Review Comment:
Thanks @shehabgamin for your review, may I suggest to keep in mind also a future lambda reduce/fold for clear and explicit namings distinguishing the "current" iterator over the array and the "accumulator" to refer in queries? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2982634042
##########
datafusion/functions-nested/src/array_transform.rs:
##########
Review Comment:
Done at https://github.com/apache/datafusion/pull/18921/changes/6ae73cb4a569d6693d37b601cca38a4c689722df, thanks for the tests
> 1. replace null lists that is not empty underneath, with nulls with empty list underneath.
...
I'm conflicted on whether we should fix case 1 for them as it is costly and the user might have prior knowledge to avoid that
I was waiting your input whether this should be handled or not. I share your concern regarding the cost of this, but also agree that it should be fixed by default, as done now. I think it can be made configurable, either via a config like `SET array_transform.remove_nulls = false` or an optional argument, and add an UDF that does that cleanup so an array can be explicit cleaned once and then be transformed multiples times across a plan without being re-cleaned again for each transformation, but not on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2996271140
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
Review Comment:
> I kept value_fields_with_lambda_udf working with the ValueOrLambda enum because it needs to check the number of all arguments for LambdaTypeSignature::Any(usize) and if/when we add TypeSignature::Exact equivalent it will likely define the exact position of lambdas, which will require full positional info
yeah I can't think of another way of removing `ValueOrLambda` from `value_fields_with_lambda_udf` and still verify that we have the correct positional info.
> Just one minor thing to note is that both changes at coerce_value_types and lambas_parameters postpone the detection of wrong parameters positions to return_field_from_args, which it's not ideal but it's not a big deal either
An error would happen regardless, just caught by Df internally rather than the implementor right? imo I think it's fine since validating argument ordering is a bit out of scope for `coerce_value_types` or `lambas_parameters` anyway. If we see a lambda function in the future that needs strict position validation, we could expose a method in the trait for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3644747290
@gstvg This is super exciting! I'll review over the next couple of days.
cc @SparkApplicationMaster @andygrove would be great to get your thoughts too 😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3088598362
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
@rluvaton @LiaCastaneda done at https://github.com/apache/datafusion/pull/18921/changes/ca260a7ef6623251705f0de7d2e93f6affd5ab89
It also include minor typo fixes and changing physical LambdaExpr data_type/nullable to match that of logical lambda, which I forgot to do in https://github.com/apache/datafusion/pull/18921#pullrequestreview-4038814640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3654993187
This is ready for a first look @shehabgamin :rocket:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3064357703
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
It does seem that the contract of `get_type` is to return the return type no? (at least going by the boolean expressions that always return `bool`)? imo I think it's more honest to return `DataType::Null `than having it return the body type, also because as @pepijnve mentions, there is no actual Arrow type for a lambda. At the call site you'd then be more explicit with handling the body type: `lambda.body.to_field(schema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3065458277
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -199,6 +203,13 @@ impl ExprSchemable for Expr {
// Grouping sets do not really have a type and do not appear in projections
Ok(DataType::Null)
}
+ Expr::HigherOrderFunction(_func) => {
+ Ok(self.to_field(schema)?.1.data_type().clone())
+ }
+ Expr::Lambda(Lambda { params: _, body }) => body.get_type(schema),
Review Comment:
Agreed, done at https://github.com/apache/datafusion/pull/18921/changes/96d8ad2f8f4d77b947b59877768f577e96788b22, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3080881683
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
the link have problem because of the large pr and the collapsed file, could you please share a different one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3075128457
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
Based on what you said on https://github.com/apache/datafusion/issues/21231#issuecomment-4188619900 , if the `LambdaExpr` get's wrapped with a debug expression that print the input and output, or add timing, then lambda evaluation will break at http://github.com/apache/datafusion/pull/18921/changes#diff-82f03654507864698a026045144642b3dcda8aedc20d29aea782f28b47b9d0eaR245
What I imagine could fix:
1. Since those wrapper expressions should likely return only the wrapped expression as children(in this case the lambda), we could repeatedly call `expr.children()` until we find a `LambdaExpr`, as long as every call return a single children, otherwise return an error. And then wrap the lambda *body* with those wrappers to make sure they actually get called
2. Get lambdas positions on `HigherOrderFunction::try_new` (but not on `HigherOrderFunction::new` as it called during tree rewrites and a wrapped lambda during one would be missed). Then we can check if an arg is a lambda or not by it's position and not via downcasting, and evaluate the lambda as a regular expr via it's `PhysicalExpr::evaluate` somehow
3. Add `as_inner_any` method to `PhysicalExpr` which by default returns `as_any` but that should return `as_inner_any` for "wrapper" expressions
4. Document that "wrapper" expressions `as_any` should return the wrapped expressionns instead of itself
5. Document that `LambdaExpr` should not be wrapped, and it's body should be wrapped instead
WDYT?
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
Based on what you said on https://github.com/apache/datafusion/issues/21231#issuecomment-4188619900 , if the `LambdaExpr` get's wrapped with a debug expression that print the input and output, or add timing, then lambda evaluation will break at http://github.com/apache/datafusion/pull/18921/changes#diff-82f03654507864698a026045144642b3dcda8aedc20d29aea782f28b47b9d0eaR245
What I imagine could fix:
1. Since those wrapper expressions should likely return only the wrapped expression as children(in this case the lambda), we could repeatedly call `expr.children()` until we find a `LambdaExpr`, as long as every call return a single children, otherwise return an error. And then wrap the lambda *body* with those wrappers to make sure they actually get called
2. Get lambdas positions on `HigherOrderFunction::try_new` (but not on `HigherOrderFunction::new` as it called during tree rewrites and a wrapped lambda during one would be missed). Then we can check if an arg is a lambda or not by it's position and not via downcasting, and evaluate the lambda as a regular expr via it's `PhysicalExpr::evaluate` somehow
3. Add `as_inner_any` method to `PhysicalExpr` which by default returns `as_any` but that should return `as_inner_any` for "wrapper" expressions
4. Document that "wrapper" expressions `as_any` should return the wrapped expressionns instead of itself
5. Document that `LambdaExpr` should not be wrapped, and it's body should be wrapped instead
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3073455172
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
Review Comment:
nit, we can avoid the clone if the nullability is the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074576663
##########
datafusion/physical-expr/src/expressions/lambda_variable.rs:
##########
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Physical lambda variable reference: [`LambdaVariable`]
+
+use std::any::Any;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::physical_expr::PhysicalExpr;
+use arrow::datatypes::FieldRef;
+use arrow::{
+ datatypes::{DataType, Schema},
+ record_batch::RecordBatch,
+};
+
+use datafusion_common::{Result, internal_err, plan_err};
+use datafusion_expr::ColumnarValue;
+
+/// Represents the lambda variable with a given name and field
+#[derive(Debug, Clone)]
+pub struct LambdaVariable {
+ name: String,
Review Comment:
included at https://github.com/apache/datafusion/pull/18921/changes/2259f70e727d61b8e0e02f01556635fedf694891
I copied from the logical counterpart (where I think we should make the field optional as in Expr::Placeholder, and keep a dedicated name ) without realizing we could use the field name here, thanks
##########
datafusion/expr/src/type_coercion/functions.rs:
##########
@@ -148,6 +151,81 @@ pub fn fields_with_udf<F: UDFCoercionExt>(
.collect())
}
+/// Performs type coercion for higher order function arguments.
+///
+/// For value arguments, returns the field to which each
+/// argument must be coerced to match `signature`.
+/// For lambda arguments, returns a clone of the associated data
+///
+/// For more details on coercion in general, please see the
+/// [`type_coercion`](crate::type_coercion) module.
+pub fn value_fields_with_higher_order_udf<L: Clone>(
Review Comment:
https://github.com/apache/datafusion/pull/18921/changes/474b22ec42bfa62531cc4072accb4b9b5632e472 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "linhr (via GitHub)" <gi...@apache.org>.
linhr commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3927881842
Sorry for missing the update on this thread and thanks for the detailed analysis @gstvg!
The reasoning makes a lot of sense to me. And it's great to see your investigation on the Spark codebase.
I didn't think deeply enough about how the lambda function would need to interact with various parts in DataFusion, such as tree traversal, SQL unparser, and the general ability to preserve argument positions. So the `LambdaUDF` trait I mentioned indeed has many limitations.
Therefore, feel free to continue the ideas you are pursuing! And let's see if DataFusion maintainers who are familiar with this part of the codebase have anything to add for the approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3946052171
Thanks @rluvaton and @gstvg , its nice you mentioned `array_transform`, the tricky part for this function is its return type depends on lambda
```
array_transform(array<T>, function<T, U>) -> array<U>
```
> I want to keep the simplicity of ScalarUDF which means that in order to evaluate a lambda expression I don't need to construct stuff, only need to provide the input and maybe some options for future use.
Right, on high level it could be like
```
pub struct LambdaExpr {
/// Parameter names/types already resolved
pub param_types: Vec<DataType>,
/// Expression body, what needs to be evaluated, this thing potentially can be UDF
pub body: Arc<dyn PhysicalExpr>,
}
```
Impl
```
impl LambdaExpr {
pub fn new(
param_types: Vec<DataType>,
body: Arc<dyn PhysicalExpr>,
) -> Self {
Self { param_types, body }
}
/// Evaluate lambda over provided arrays
pub fn evaluate_with_args(
&self,
args: Vec<ArrayRef>,
) -> Result<ArrayRef> {
// Build synthetic schema
let fields: Vec<Field> = self.param_types
.iter()
.enumerate()
.map(|(i, dt)| Field::new(format!("arg{}", i), dt.clone(), true))
.collect();
let schema = Arc::new(Schema::new(fields));
let batch = RecordBatch::try_new(schema, args)?;
self.body.evaluate(&batch) // this where our UDF would be called
}
}
```
So for example `x -> x + 1` we need to parse expression and create our Lambda, so we need to modify parser to get structures below from user defined code and there is an existing ticket https://github.com/apache/datafusion-sqlparser-rs/issues/1273
```
// Parameter x at column 0
let x = Arc::new(ColumnExpr::new(0));
// Literal 1
let one = Arc::new(LiteralExpr::new(
ScalarValue::Int32(Some(1))
));
// x + 1
let body = Arc::new(BinaryExpr::new(
x,
one,
Operator::Add,
));
// Lambda(x) -> x + 1
let lambda = LambdaExpr::new(
vec![DataType::Int32],
body,
);
```
and call it from caller built in function
```
fn array_transform(
list_array: &ListArray,
lambda: &LambdaExpr,
) -> Result<ListArray> {
let values = list_array.values().clone();
// evaluate lambda on flattened child array
let transformed =
lambda.evaluate_with_args(vec![values])?;
Ok(ListArray::new(
list_array.data_type().clone(),
list_array.offsets().clone(),
transformed,
list_array.nulls().cloned(),
))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3956255462
> But I would really appreciate if you reviewed it, I can push a `DOC.md` to this branch and we can start a review on it to not add even more comments here, and then finally update the PR description, WDYT?
it sounds great IMO, there are a lot of people expressing the interest to the topic and love to review as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4025410602
looks like latest substrait is not backward compatible (for a patch change) and its breaking 52.2 builds #20756 @benbellick . if you agree perhaps we should pin version of substrat to previous one in datafusion for df 53 release ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2924724943
##########
DOC.md:
##########
@@ -0,0 +1,1166 @@
+This PR adds support for lambdas with column capture and the `array_transform` function used to test the lambda implementation. Example usage:
+
+```sql
+CREATE TABLE t as SELECT 2 as n;
+
+SELECT array_transform([2, 3], v -> v != t.n) from t;
+
+[false, true]
+
+-- arbitrally nested lambdas are also supported
+SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));
+
+[[[4, 6]]]
+```
+
+Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code
+
+3 new `Expr` variants are added, `LambdaFunction`, owing a new trait `LambdaUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. The reasoning why not using `Column` instead is later on this doc.
+
+Their logical representations:
+
+```rust
+enum Expr {
+ LambdaFunction(LambdaFunction), // array_transform([2, 3], v -> v != t.n)
+ Lambda(Lambda), // v -> v != t.n
+ LambdaVariable(LambdaVariable), // v, of the lambda body: v != t.n
+ ...
+}
+
+// array_transform([2, 3], v -> v != t.n)
+struct LambdaFunction {
+ pub func: Arc<dyn LambdaUDF>, // global instance of array_transform
+ pub args: Vec<Expr>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != n)]
+}
+
+// v -> v != t.n
+struct Lambda {
+ pub params: Vec<String>, // ["v"]
+ pub body: Box<Expr>, // v != n
+}
+
+// v, of the lambda body: v != t.n
+struct LambdaVariable {
+ pub name: String, // "v"
+ pub field: Option<FieldRef>, // Some(Field::new("", DataType::Int32, false))
+ pub spans: Spans,
+}
+
+```
+
+The example would be planned into a tree like this:
+
+```
+LambdaFunctionExpression
+ name: array_transform
+ children:
+ 1. ListExpression [2,3]
+ 2. LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression (!=)
+ left:
+ LambdaVariableExpression("v", Some(Field::new("", Int32, false)))
+ right:
+ ColumnExpression("t.n")
+```
+
+The physical counterparts definition:
+
+```rust
+
+struct LambdaFunctionExpr {
+ fun: Arc<dyn LambdaUDF>, // global instance of array_transform
+ name: String, // "array_transform"
+ args: Vec<Arc<dyn PhysicalExpr>>, // [LiteralExpr([2, 3], LambdaExpr("v -> v != t.n"))]
+ return_field: FieldRef, // Field::new("", DataType::new_list(DataType::Boolean, false), false)
+ config_options: Arc<ConfigOptions>,
+}
+
+
+struct LambdaExpr {
+ params: Vec<String>, // ["v"]
+ body: Arc<dyn PhysicalExpr>, // v -> v != t.n
+}
+
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ value: Option<ColumnarValue>, // reasoning later on
+}
+```
+
+Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `LambdaUDF` and the `array_transform` implementation of `LambdaUDF` relevant methods, collapsed due to their size
+
+<details><summary>Physical planning implementation is trivial:</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let physical_args =
+ create_physical_exprs(args, input_dfschema, execution_props)?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ Expr::Lambda(Lambda { params, body }) => Ok(Arc::new(LambdaExpr::new(
+ params.clone(),
+ create_physical_expr(body, input_dfschema, execution_props)?,
+ ))),
+ Expr::LambdaVariable(LambdaVariable {
+ name,
+ field,
+ spans: _,
+ }) => lambda_variable(
+ name,
+ Arc::clone(field),
+ ),
+ }
+}
+```
+
+</details>
+<br>
+
+The added `LambdaUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of:
+1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values
+2. the addition of `lambdas_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments
+3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`.
+
+<details><summary>LambdaUDF</summary>
+
+```rust
+
+trait LambdaUDF {
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ /// based on the Field of the non-lambda arguments
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2, 8], v -> v > 4)`
+ ///
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false)))]), // the Field associated with the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
+ /// ]?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// None, // it's a value, return None
+ /// // it's a lambda, return it's supported parameters, regardless of how many are actually used
+ /// Some(vec![
+ /// Field::new("", DataType::Int32, false), // the value being transformed,
+ /// Field::new("", DataType::Int32, false), // the 1-based index being transformed, not used on the example above, but implementations doesn't need to care about it
+ /// ])
+ /// ]
+ /// )
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>>;
+ fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue>;
+ // ... omitted methods that are similar in ScalarUDFImpl
+}
+
+pub enum ValueOrLambdaParameter {
+ /// A columnar value with the given field
+ Value(FieldRef),
+ /// A lambda
+ Lambda,
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field returned by the
+ /// lambda when executed with the arguments returned from `LambdaUDF::lambdas_parameters`
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ // ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ // ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false))
+ // ]`
+ pub arg_fields: &'a [ValueOrLambdaField],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `my_function(column_a, 5)`
+ /// this field will be `[None, Some(ScalarValue::Int32(Some(5)))]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// A tagged FieldRef indicating whether it correspond the field of a value or the field of the output of a lambda argument
+pub enum ValueOrLambdaField {
+ /// The FieldRef of a ColumnarValue argument
+ Value(FieldRef),
+ /// The return FieldRef of the lambda body when evaluated with the parameters from LambdaUDF::lambda_parameters
+ Lambda(FieldRef),
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments to the function
+ pub args: Vec<ValueOrLambda>,
+ /// Field associated with each arg, if it exists
+ pub arg_fields: Vec<ValueOrLambdaField>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned (from `return_type`
+ /// or `return_field_from_args`) when creating the physical expression
+ /// from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+/// A lambda argument to a LambdaFunction
+pub struct LambdaFunctionLambdaArg {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ pub params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ pub body: Arc<dyn PhysicalExpr>,
+ /// A RecordBatch containing at least the captured columns inside this lambda body, if any
+ /// Note that it may contain additional, non-specified columns,
+ /// but that's implementation detail and should not be relied upon
+ ///
+ /// For example, for `array_transform([2], v -> v + t.a + t.b)`,
+ /// this will be a `RecordBatch` with at least two columns, `t.a` and `t.b`
+ pub captures: Option<RecordBatch>,
+}
+
+// An argument to a LambdaUDF
+pub enum ValueOrLambda {
+ Value(ColumnarValue),
+ Lambda(LambdaFunctionLambdaArg),
+}
+```
+
+
+</details>
+
+<details><summary>array_transform lambdas_parameters implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambdaParameter],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ // list is the field of [2, 3]: Field::new("", DataType::new_list(DataType::Int32, false), false)
+ let [ValueOrLambdaParameter::Value(list), ValueOrLambdaParameter::Lambda] = args
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // the field of [2, 3] inner values: Field::new("", DataType::Int32, false)
+ let (field, index_type) = match list.data_type() {
+ DataType::List(field) => (field, DataType::Int32),
+ DataType::LargeList(field) => (field, DataType::Int64),
+ DataType::FixedSizeList(field, _) => (field, DataType::Int32),
+ _ => return exec_err!("expected list, got {list}"),
+ };
+
+ // we don't need to omit the index in the case the lambda don't specify, e.g. array_transform([], v -> v*2),
+ // nor check whether the lambda contains more than two parameters, e.g. array_transform([], (v, i, j) -> v+i+j),
+ // as datafusion will do that for us
+ let value = Field::new("", field.data_type().clone(), field.is_nullable())
+ .with_metadata(field.metadata().clone());
+ let index = Field::new("", index_type, false);
+
+ Ok(vec![None, Some(vec![value, index])])
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform return_field_from_args implementation</summary>
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn return_field_from_args(
+ &self,
+ args: datafusion_expr::LambdaReturnFieldArgs,
+ ) -> Result<Arc<Field>> {
+ // [
+ // Field::new("", DataType::new_list(DataType::Int32, false), false),
+ // Field::new("", DataType::Boolean, false),
+ // ]
+ let [ValueOrLambdaField::Value(list), ValueOrLambdaField::Lambda(lambda)] =
+ take_function_args(self.name(), args.arg_fields)?
+ else {
+ return exec_err!(
+ "{} expects a value follewed by a lambda, got {:?}",
+ self.name(),
+ args
+ );
+ };
+
+ // lambda is the return_field of the lambda body
+ // when evaluated with the parameters from lambdas_parameters
+ let field = Arc::new(Field::new(
+ Field::LIST_FIELD_DEFAULT_NAME,
+ lambda.data_type().clone(),
+ lambda.is_nullable(),
+ ));
+
+ let return_type = match list.data_type() {
+ DataType::List(_) => DataType::List(field),
+ DataType::LargeList(_) => DataType::LargeList(field),
+ DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
+ other => plan_err!("expected list, got {other}"),
+ };
+
+ Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
+ }
+}
+```
+
+</details>
+
+<details><summary>array_transform invoke_with_args implementation</summary>
+
+
+```rust
+impl LambdaUDF for ArrayTransform {
+ fn invoke_with_args(&self, args: LambdaFunctionArgs) -> Result<ColumnarValue> {
+ let [list_value, lambda] = take_function_args(self.name(), &args.args)?;
+
+ // list = [2, 3]
+ // lambda = LambdaFunctionLambdaArg {
+ // params: vec![Field::new("v", DataType::Int32, false)],
+ // body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ // captures: Some(record_batch!("t.n", Int32, [2]))
+ // }
+ let (ValueOrLambda::Value(list_value), ValueOrLambda::Lambda(lambda)) =
+ (list_value, lambda)
+ else {
+ return exec_err!(
+ "{} expects a value followed by a lambda, got {} and {}",
+ self.name(),
+ list_value,
+ lambda,
+ );
+ };
+
+ let list_array = list_value.to_array(args.number_rows)?;
+ let list_values = match list_array.data_type() {
+ DataType::List(_) => list_array.as_list::<i32>().values(),
+ DataType::LargeList(_) => list_array.as_list::<i64>().values(),
+ DataType::FixedSizeList(_, _) => list_array.as_fixed_size_list().values(),
+ other => exec_err!("expected list, got {other}")
+ }
+
+ // if any column got captured, we need to adjust it to the values arrays,
+ // duplicating values of list with mulitple values and removing values of empty lists
+ // list_indices is not cheap so is important to avoid it when no column is captured
+ let adjusted_captures = lambda
+ .captures
+ .as_ref()
+ //list_indices return the row_number for each sublist element: [[1, 2], [3], [4]] => [0,0,1,2], not included here
+ .map(|captures| take_record_batch(captures, &list_indices(&list_array)?))
+ .transpose()?
+ .unwrap_or_else(|| {
+ RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(list_values.len())),
+ )
+ .unwrap()
+ });
+
+ // by using closures, bind_lambda_variables can evaluate only the needed ones avoiding unnecessary computations
+ let values_param = || Ok(Arc::clone(list_values));
+ //elements_indices return the index of each element within its sublist: [[5, 3], [7, 1, 1]] => [1, 2, 1, 2, 3], not included here
+ let indices_param = || elements_indices(&list_array);
+
+ let binded_body = bind_lambda_variables(
+ Arc::clone(&lambda.body),
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = binded_body
+ .evaluate(&adjusted_captures)?
+ .into_array(list_values.len())?;
+
+ let field = match args.return_field.data_type() {
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::FixedSizeList(field, _) => Arc::clone(field),
+ _ => {
+ return exec_err!(
+ "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
+ self.name(),
+ args.return_field
+ )
+ }
+ };
+
+ let transformed_list = match list_array.data_type() {
+ DataType::List(_) => {
+ let list = list_array.as_list();
+
+ Arc::new(ListArray::new(
+ field,
+ list.offsets().clone(),
+ transformed_values,
+ list.nulls().cloned(),
+ )) as ArrayRef
+ }
+ DataType::LargeList(_) => {
+ let large_list = list_array.as_list();
+
+ Arc::new(LargeListArray::new(
+ field,
+ large_list.offsets().clone(),
+ transformed_values,
+ large_list.nulls().cloned(),
+ ))
+ }
+ DataType::FixedSizeList(_, value_length) => {
+ Arc::new(FixedSizeListArray::new(
+ field,
+ *value_length,
+ transformed_values,
+ list_array.as_fixed_size_list().nulls().cloned(),
+ ))
+ }
+ other => exec_err!("expected list, got {other}")?,
+ };
+
+ Ok(ColumnarValue::Array(transformed_list))
+ }
+}
+```
+
+</details>
+
+<details><summary>How relevant LambdaUDF methods would be called and what they would return during planning and evaluation of the example</summary>
+
+
+```rust
+// this is called at sql planning
+let lambdas_parameters = lambda_udf.lambdas_parameters(&[
+ ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), // the Field of the [2, 3] literal
+ ValueOrLambdaParameter::Lambda, // A unspecified lambda. On the example, v -> v != t.n
+])?;
+
+assert_eq!(
+ lambdas_parameters,
+ vec![
+ // the [2, 3] argument, not a lambda so no parameters
+ None,
+ // the parameters that *can* be declared on the lambda, and not only
+ // those actually declared: the implementation doesn't need to care
+ // about it
+ Some(vec![
+ Field::new("", DataType::Int32, false), // the list inner value
+ Field::new("", DataType::Int32, false), // the 1-based index of the element being transformed
+ ])]
+);
+
+
+
+// this is called every time ExprSchemable is called on a LambdaFunction
+let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
+ arg_fields: &[
+ ValueOrLambdaField::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
+ ValueOrLambdaField::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != t.n" when "v" is of the type returned in lambdas_parameters
+ ],
+ scalar_arguments // irrelevant
+})?;
+
+assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));
+
+
+
+let value = array_transform.evaluate(&LambdaFunctionArgs {
+ args: vec![
+ ValueOrLambda::Value(List([2, 3])),
+ ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params: vec![Field::new("v", DataType::Int32, false)],
+ body: PhysicalExpr("v != t.n"),// the physical expression of the lambda *body*, and not the lambda itself: this is not a LambdaExpr.
+ captures: Some(record_batch!("t.n", Int32, [2]))
+ }),
+ ],
+ arg_fields, // same as above
+ number_rows: 1,
+ return_field, // same as above
+ config_options, // irrelevant
+})?;
+
+assert_eq!(value, BooleanArray::from([false, true]))
+```
+
+</details>
+<br>
+<br>
+
+A pair LambdaUDF/LambdaUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) #8045
+
+LambdaFunction invocation:
+
+Instead of evaluating all it's arguments as ScalarFunction, LambdaFunction does the following:
+
+1. If it's a non lambda argument, evaluate as usual, and provide the resulting `ColumnarValue` to `LambdaUDF::evaluate` as a `ValueOrLambda::Value`
+2. If it's a lambda, construct a `LambdaFunctionLambdaArg` containing the lambda body physical expression and a record batch containing any captured columns as a `ValueOrLambda::Lambda` and provide it to `LambdaUDF::evaluate`. To avoid costly copies of uncaptured columns, we swap them with a `NullArray` while keeping the number of columns on the batch the same so captured columns indices are kept stable across the whole tree. The recent #18329 instead projects-out uncaptured columns and rewrites the expr adjusting columns indexes. If that is preferrable we can generalize that implementation and use it here too.
+
+<details><summary>LambdaFunction evalution</summary>
+
+```rust
+
+impl PhysicalExpr for LambdaFunctionExpr {
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let args = self.args
+ .map(|arg| {
+ match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ // helper method that returns the indices of the captured columns. In the example, the only column available (index 0) is captured, so this would be HashSet(0)
+ let captures = lambda.captures();
+
+ let captures = if !captures.is_empty() {
+ let (fields, columns): (Vec<_>, _) = std::iter::zip(
+ batch.schema_ref().fields(),
+ batch.columns(),
+ )
+ .enumerate()
+ .map(|(column_index, (field, column))| {
+ if captures.contains(&column_index) {
+ (Arc::clone(field), Arc::clone(column))
+ } else {
+ (
+ Arc::new(Field::new(
+ field.name(),
+ DataType::Null,
+ false,
+ )),
+ Arc::new(NullArray::new(column.len())) as _,
+ )
+ }
+ })
+ .unzip();
+
+ let schema = Arc::new(Schema::new(fields));
+
+ Some(RecordBatch::try_new(schema, columns)?)
+ } else {
+ None
+ };
+
+ Ok(ValueOrLambda::Lambda(LambdaFunctionLambdaArg {
+ params, // irrelevant,
+ body: Arc::clone(lambda.body()), // use the lambda body and not the lambda itself
+ captures,
+ }))
+ }
+ None => Ok(ValueOrLambda::Value(arg.evaluate(batch)?)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(LambdaFunctionArgs {
+ args,
+ arg_fields, // irrelevant
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ Ok(output)
+ }
+}
+
+```
+
+</details>
+<br>
+
+Why `LambdaVariable` and not `Column`:
+
+Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t".
+
+Example of code of another traversal that would break:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // if this is a lambda column, this function will break
+ used_columns.insert(col.index());
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ ...
+}
+```
+
+Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters.
+
+By including a `FieldRef` on `LambdaVariable` that should be resolved either during construction time, as in the sql planner, or later by the an `AnalyzerRule`, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field:
+
+<details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary>
+
+```rust
+impl ExprSchemable for Expr {
+ fn to_field(
+ &self,
+ schema: &dyn ExprSchema,
+ ) -> Result<(Option<TableReference>, Arc<Field>)> {
+ let (relation, schema_name) = self.qualified_name();
+ let field = match self {
+ Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field.ok_or_else(|| plan_err!("Unresolved LambdaVariable {}", l.name)))),
+ ...
+ }?;
+
+ Ok((
+ relation,
+ Arc::new(field.as_ref().clone().with_name(schema_name)),
+ ))
+ }
+ ...
+}
+
+impl PhysicalExpr for LambdaVariable {
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.field))
+ }
+ ...
+}
+```
+
+</details>
+<br>
+
+For reference, [Spark](https://github.com/apache/spark/blob/8b68a172d34d2ed9bd0a2deefcae1840a78143b6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala#L77) and [Substrait](https://substrait.io/expressions/lambda_expressions/#parameter-references) also use a specialized node instead of a regular column
+
+There's also discussions on making every expr own it's type: #18845, #12604
+
+<details><summary>Possible fixes discarded due to complexity, requiring downstream changes and implementation size:</summary>
+
+1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR.
+
+```rust
+impl Expr {
+ pub fn transform_with_lambdas_params<
+ F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
+ >(
+ self,
+ mut f: F,
+ ) -> Result<Transformed<Self>> {}
+}
+```
+
+How minimize_join_filter would looks like:
+
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply_with_lambdas_params(|expr, lambdas_params| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !lambdas_params.contains(col.name()) {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream.
+
+```rust
+//logical
+struct Column {
+ pub relation: Option<TableReference>,
+ pub name: String,
+ pub spans: Spans,
+ pub is_lambda_parameter: bool,
+}
+
+//physical
+struct Column {
+ name: String,
+ index: usize,
+ is_lambda_parameter: bool,
+}
+```
+
+
+How minimize_join_filter would look like:
+
+```rust
+fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
+ let mut used_columns = HashSet::new();
+ expr.apply(|expr| {
+ if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+ // dont include lambdas parameters
+ if !col.is_lambda_parameter {
+ used_columns.insert(col.index());
+ }
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })
+ ...
+}
+```
+
+
+1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed:
+
+```rust
+impl Expr {
+ pub fn transform_with_schema<
+ F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
+ >(
+ self,
+ schema: &DFSchema,
+ f: F,
+ ) -> Result<Transformed<Self>> { ... }
+ ... other methods
+}
+```
+
+For any given LambdaFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from LambdaUDF::lambdas_parameters
+How it would look like:
+
+```rust
+
+pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
+ let mut has_placeholder = false;
+ // Provide the schema as the first argument.
+ // Transforming closure receive an adjusted_schema as argument
+ self.transform_with_schema(schema, |mut expr, adjusted_schema| {
+ match &mut expr {
+ // Default to assuming the arguments are the same type
+ Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
+ // use adjusted_schema and not schema. Those expressions may contain
+ // columns referring to a lambda parameter, which Field would only be
+ // available in adjusted_schema and not in schema
+ rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
+ rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
+ }
+ ....
+
+```
+
+2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned
+
+
+```rust
+impl LogicalPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+trait ExecutionPlan {
+ fn lambda_extended_schema(&self) -> &DFSchema;
+}
+
+//usage
+impl LogicalPlan {
+ pub fn replace_params_with_values(
+ self,
+ param_values: &ParamValues,
+ ) -> Result<LogicalPlan> {
+ self.transform_up_with_subqueries(|plan| {
+ // use plan.lambda_extended_schema() containing lambdas parameters
+ // instead of plan.schema() which wont
+ let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
+ let name_preserver = NamePreserver::new(&plan);
+ plan.map_expressions(|e| {
+ // if this expression is child of lambda and contain columns referring it's parameters
+ // the lambda_extended_schema already contain them
+ let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
+ ....
+
+```
+</details>
+<br>
+
+`LambdaVariable` evaluation, current implementation:
+
+The physical `LambdaVariable` contains an optional `ColumnarValue` that must be binded for each batch before evaluation with the helper function `bind_lambda_variables`, which rewrites the whole lambda body, binding any variable of the tree.
+
+<details><summary>LambdaVariable::evaluate</summary>
+
+```rust
+impl PhysicalExpr for LambdaVariable {
+ fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
+ self.value.clone().ok_or_else(|| exec_datafusion_err!("Physical LambdaVariable {} unbinded value", self.name))
+ }
+}
+```
+
+</details>
+<br>
+
+Unbinded:
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), None)
+ right:
+ ColumnExpression("n")
+```
+
+After binding:
+
+```
+LambdaExpression
+ parameters: ["v"]
+ body:
+ ComparisonExpression(!=)
+ left:
+ LambdaVariableExpression("v", Field::new("", Int32, false), Some([2, 3]))
+ right:
+ ColumnExpression("n")
+```
+
+Alternative:
+
+Make the `LambdaVariable` evaluate it's value from the batch passed to `PhysicalExpr::evaluate` as a regular column. For that, instead of binding the body, the `LambdaUDF` implementation would merge the captured batch of a lambda with the values of it's parameters. So that it happen via an index as a regular column, the schema used plan to physical `LambdaVariable` must contain the lambda parameters. This would be the only place during planning that a schema would contain those parameters. Otherwise it only can get the value from the batch via name instead of index
+
+1. Add a index to LambdaVariable, similar to Column, and remove the optional value.
+
+```rust
+struct LambdaVariable {
+ name: String, // "v", of the lambda body: v != t.n
+ field: FieldRef, // Field::new("", DataType::Int32, false)
+ index: usize, // 1
+}
+```
+
+2. Insert the lambda parameters only at the Schema used to do the physical planning, to compute the index of a LambdaVariable
+
+<details><summary>how physical planning would look like</summary>
+
+```rust
+fn create_physical_expr(
+ e: &Expr,
+ input_dfschema: &DFSchema,
+ execution_props: &ExecutionProps,
+) -> Result<Arc<dyn PhysicalExpr>> {
+ let input_schema = input_dfschema.as_arrow();
+
+ match e {
+ ...
+ Expr::LambdaFunction(LambdaFunction { func, args}) => {
+ let args_metadata = args.iter()
+ .map(|arg| if arg.is::<LambdaExpr>() {
+ Ok(ValueOrLambdaParameter::Lambda)
+ } else {
+ Ok(ValueOrLambdaParameter::Value(arg.to_field(input_dfschema)?))
+ })
+ .collect()?;
+
+ let lambdas_parameters = func.lambdas_parameters(&args_metadata)?;
+
+ let physical_args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambda_parameters)| {
+ match (arg.downcast_ref::<LambdaExpr>(), lambda_parameters) {
+ (Some(lambda), Some(lambda_parameters)) => {
+ let extended_dfschema = merge_schema_and_parameters(input_dfschame, lambda_parameters)?;
+
+ create_physical_expr(body, extended_dfschema, execution_props)
+ }
+ (None, None) => create_physical_expr(arg, input_dfschema, execution_props),
+ (Some(_), None) => plan_err!("lambdas_parameters returned None for a lambda")
+ (None, Some(_)) => plan_err!("lambdas_parameters returned Some for a non lambda")
+ }
+ })
+ .collect()?;
+
+ Ok(Arc::new(LambdaFunctionExpr::try_new(
+ Arc::clone(func),
+ physical_args,
+ input_schema,
+ config_options: ... // irrelevant
+ )?))
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+3. Insert the lambda parameters values into the RecordBatch during the evaluation phase: the LambdaUDF, instead of binding the lambda body variables, inserts it's parameters on the captured RecordBatch it receives on LambdaFunctionLambdaArg.
+
+How ArrayTransform::invoke_with_args would look like:
+
+```rust
+ ...
+ let values_param = || Ok(Arc::clone(list_values));
+ let indices_param = || elements_indices(&list_array);
+
+ let merged_batch = merge_captures_with_params(
+ adjusted_captures,
+ &lambda.params,
+ &[&values_param, &indices_param],
+ )?;
+
+ // call the transforming expression with the record batch
+ let transformed_values = lambda.body
+ .evaluate(&merged_batch)?
+ .into_array(list_values.len())?;
+
+ ...
+```
+
+<br>
+
+Why is `LambdaVariable` `Field` is an `Option`?
+
+So expr_api users can construct a LambdaVariable just by using it's name, without having to set it's field. An `AnalyzerRule` will then set the `LambdaVariable` field based on the returned values from `LambdaUDF::lambdas_parameters` of any `LambdaFunction` it finds while traversing down a expr tree. We may include that rule on the default rules list for when the plan/expression tree is transformed by another rule in a way that changes the types of non lambda arguments of a lambda function, as it may change the types of it's lambda parameters, which would render `LambdaVariable` field's out of sync, as the rule would fix it. Or to not increase planning time we don't include it by default and instruct `expr_api` users to add it manually if needed
+
+
+
+```rust
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value")
+ )
+)
+
+//instead of
+
+array_transform(
+ col("my_array"),
+ lambda(
+ vec!["current_value"],
+ 2 * lambda_variable("current_value", Field::new("", DataType::Int32, false))
+ )
+)
+```
+
+
+Why set `LambdaVariable` field during sql planning if it's optional and can be set later via an `AnalyzerRule`?
+
+Some parts of sql planning checks the type/nullability of the already planned children expression of the expr it's planning, and would error if doing so on a unresolved `LambdaVariable`
+Take as example this expression: `array_transform([[0, 1]], v -> v[1])`. `FieldAccess` `v[1]` planning is handled by the `ExprPlanner` `FieldAccessPlanner`, which checks the datatype of `v`, a lambda variable, which `ExprSchemable` implementation depends on it's field being resolved, and not on the `PlannerContext` schema, requiring sql planner to plan `LambdaVariables` with a resolved field
+
+
+<details><summary>FieldAccessPlanner</summary>
+
+```rust
+pub struct FieldAccessPlanner;
+
+impl ExprPlanner for FieldAccessPlanner {
+ fn plan_field_access(
+ &self,
+ expr: RawFieldAccessExpr, // "v[1]"
+ schema: &DFSchema,
+ ) -> Result<PlannerResult<RawFieldAccessExpr>> {
+ // { "v", "[1]" }
+ let RawFieldAccessExpr { expr, field_access } = expr;
+
+ match field_access {
+ ...
+ // expr[idx] ==> array_element(expr, idx)
+ GetFieldAccess::ListIndex { key: index } => {
+ match expr {
+ ...
+ // ExprSchemable::get_type called
+ _ if matches!(expr.get_type(schema)?, DataType::Map(_, _)) => {
+ Ok(PlannerResult::Planned(Expr::ScalarFunction(
+ ScalarFunction::new_udf(
+ get_field_inner(),
+ vec![expr, *index],
+ ),
+ )))
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+</details>
+<br>
+
+ Therefore we can't plan all arguments on a single pass, and must first plan the non-lambda arguments, collect their types and nullability, pass them to `LambdaUDF::lambdas_parameters`, which will derive the type of it's lambda parameters based on the type of it's non-lambda argument, and return it to the planner, which, for each unplanned lambda argument, will create a new `PlannerContext` via `with_lambda_parameters`, which contains a mapping of lambdas parameters names to it's type. Then, when planning a `ast::Identifier`, it first check whether a lambda parameter with the given name exists, and if so, plans it into a `Expr::LambdaVariable` with a resolved field, otherwise plan it into a regular `Expr::Column`.
+
+
+
+<details><summary>sql planning</summary>
+
+
+```rust
+struct PlannerContext {
+ /// The parameters of all lambdas seen so far
+ lambdas_parameters: HashMap<String, FieldRef>,
+ // ... omitted fields
+}
+
+impl PlannerContext {
+ pub fn with_lambda_parameters(
+ mut self,
+ arguments: impl IntoIterator<Item = FieldRef>,
+ ) -> Self {
+ self.lambdas_parameters
+ .extend(arguments.into_iter().map(|f| (f.name().clone(), f)));
+
+ self
+ }
+}
+
+// copied from sqlparser
+struct LambdaFunction {
+ pub params: OneOrManyWithParens<Ident>, // One("v")
+ pub body: Box<Expr>, // v != t.n
+}
+
+// copied from sqlparser
+enum OneOrManyWithParens<T> {
+ One(T), // "v"
+ Many(Vec<T>),
+}
+
+/// the planning would happens as the following:
+
+enum ExprOrLambda {
+ Expr(Expr), // planned [2, 3]
+ Lambda(ast::LambdaFunction), // unplanned v -> v != t.n
+}
+
+impl SqlToRel {
+ // example function, won't exist
+ fn plan_array_transform(&self, array_transform: Arc<dyn LambdaUDF>, args: Vec<ast::Expr>, schema: &DFSchema, planner_context: &mut PlannerContext) -> Result<Expr> {
+ let args = args.into_iter()
+ .map(|arg| match arg {
+ ast::Expr::LambdaFunction(l) => Ok(ExprOrLambda::Lambda(l)),//skip planning until we plan non lambda args
+ arg => Ok(ExprOrLambda::Expr(
+ self.sql_fn_arg_to_logical_expr_with_name(
+ arg,
+ schema,
+ planner_context,
+ )?,
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let args_metadata = args.iter()
+ .map(|arg| match arg {
+ Expr(expr) => Ok(ValueOrLambda::Value(expr.to_field(schema)?)),
+ Lambda(_) => Ok(ValueOrLambda::Lambda),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let lambdas_parameters = array_transform.lambdas_parameters(&args_metadata)?;
+
+ let args = std::iter::zip(args, lambdas_parameters)
+ .map(|(arg, lambdas_parameters)| match (arg, lambdas_parameters) {
+ (ExprOrLambda::Expr(planned_expr), None) => Ok(planned_expr),
+ (ExprOrLambda::Lambda(unplanned_lambda), Some(lambda_parameters)) => {
+ let params =
+ unplanned_lambda.params
+ .iter()
+ .map(|p| p.value.clone())
+ .collect();
+
+ let lambda_parameters = lambda_params
+ .into_iter()
+ .zip(¶ms)
+ .map(|(field, name)| Arc::new(field.with_name(name)));
+
+ let mut planner_context = planner_context
+ .clone()
+ .with_lambda_parameters(lambda_parameters);
+
+ Ok((
+ Expr::Lambda(Lambda {
+ params,
+ body: Box::new(self.sql_expr_to_logical_expr(
+ *lambda.body,
+ schema,
+ &mut planner_context,
+ )?),
+ }),
+ None,
+ ))
+ }
+ (ExprOrLambda::Expr(planned_expr), Some(lambda_parameters)) => plan_err!("lambdas_parameters returned Some for a value"),
+ (ExprOrLambda::Lambda(unplanned_lambda), None) => plan_err!("lambdas_parameters returned None for a lambda"),
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ Ok(Expr::LambdaFunction(LambdaFunction {
+ func: array_transform,
+ args,
+ }))
+ }
+
+ fn sql_identifier_to_expr(
+ &self,
+ id: ast::Ident,
+ schema: &DFSchema,
+ planner_context: &mut PlannerContext,
+ ) -> Result<Expr> {
+ // simplified implementation
+ if let Some(field) = planner_context.lambdas_parameters.get(id) {
+ Ok(Expr::LambdaVariable(LambdaVariable {
+ name: id, // "v"
+ field, // Field::new("", DataType::Int32, false)
+ }))
+ } else {
+ Ok(Expr::Column(Column::new(id)))
+ }
+ }
+}
+
+```
+
+</details>
+</br>
+
+`LambdaFunction` `Signature` is non functional
+
+Currenty, `LambdaUDF::signature` returns the same `Signature` as `ScalarUDF`, but it's `type_signature` field is never used, as most variants of the `TypeSignature` enum aren't applicable to a lambda, and no type coercion is applied on it's arguments, being currently a implementation responsability. We should either add lambda compatible variants to the `TypeSignature` enum, create a new `LambdaTypeSignature` and `LambdaSignature`, or support no automatic type coercion at all on lambda functions.
Review Comment:
mmm, you're right this would break with interleaved lambdas, but what if the caller [here](https://github.com/gstvg/arrow-datafusion/blob/86d5999056b49b4b597c1103ee9f57f65cacc882/datafusion/optimizer/src/analyzer/type_coercion.rs#L614) did a bit more work to identify which `DataType` corresponds to which position? That way we wouldn't need to pass the whole `ValueOrLambdaParameter` enum to `coerce_value_types` and users wouldn't need to bear the overhead of returning `Vec<Option<DataType>>` because of lambda positions.
I think this would make the API easier to use (since it mirrors `ScalarUDFImpl::coerce_types`) but if it's too much overhead to change, maybe it's fine to keep it as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2634134147
##########
datafusion/expr/src/expr.rs:
##########
@@ -398,6 +398,30 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Lambda expression
+ Lambda(Lambda),
+ LambdaColumn(LambdaColumn),
Review Comment:
Yes, done in https://github.com/apache/datafusion/pull/18921/commits/d844b2d81d9ab5c4bc95671e9134ea6475af79bb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "fbx31 (via GitHub)" <gi...@apache.org>.
fbx31 commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2628576483
##########
datafusion/expr/src/expr.rs:
##########
@@ -398,6 +398,30 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Lambda expression
+ Lambda(Lambda),
+ LambdaColumn(LambdaColumn),
Review Comment:
Thanks @shehabgamin for your review, may I suggest to keep in mind also a future lambda reduce/fold for clear and explicit namings distinguishing the "current" iterator over the array and the "accumulator" to refer in queries? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3004952892
##########
datafusion/expr/src/udlf.rs:
##########
@@ -0,0 +1,548 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`]: Lambda User Defined Functions
+
+use crate::expr::schema_name_from_exprs_comma_separated_without_space;
+use crate::{ColumnarValue, Documentation, Expr};
+use arrow::array::{ArrayRef, RecordBatch};
+use arrow::datatypes::{DataType, Field, FieldRef, Schema};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{Result, ScalarValue, not_impl_err};
+use datafusion_expr_common::dyn_eq::{DynEq, DynHash};
+use datafusion_expr_common::signature::Volatility;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::any::Any;
+use std::cmp::Ordering;
+use std::fmt::Debug;
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+/// The types of arguments for which a function has implementations.
+///
+/// [`LambdaTypeSignature`] **DOES NOT** define the types that a user query could call the
+/// function with. DataFusion will automatically coerce (cast) argument types to
+/// one of the supported function signatures, if possible.
+///
+/// # Overview
+/// Functions typically provide implementations for a small number of different
+/// argument [`DataType`]s, rather than all possible combinations. If a user
+/// calls a function with arguments that do not match any of the declared types,
+/// DataFusion will attempt to automatically coerce (add casts to) function
+/// arguments so they match the [`LambdaTypeSignature`]. See the [`type_coercion`] module
+/// for more details
+///
+/// [`type_coercion`]: crate::type_coercion
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub enum LambdaTypeSignature {
+ /// The acceptable signature and coercions rules are special for this
+ /// function.
+ ///
+ /// If this signature is specified,
+ /// DataFusion will call [`LambdaUDF::coerce_value_types`] to prepare argument types.
+ UserDefined,
+ /// One or more lambdas or arguments with arbitrary types
+ VariadicAny,
+ /// The specified number of lambdas or arguments with arbitrary types.
+ Any(usize),
+}
+
+/// Provides information necessary for calling a lambda function.
+///
+/// - [`LambdaTypeSignature`] defines the argument types that a function has implementations
+/// for.
+///
+/// - [`Volatility`] defines how the output of the function changes with the input.
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub struct LambdaSignature {
+ /// The data types that the function accepts. See [LambdaTypeSignature] for more information.
+ pub type_signature: LambdaTypeSignature,
+ /// The volatility of the function. See [Volatility] for more information.
+ pub volatility: Volatility,
+}
+
+impl LambdaSignature {
+ /// Creates a new `LambdaSignature` from a given type signature and volatility.
+ pub fn new(type_signature: LambdaTypeSignature, volatility: Volatility) -> Self {
+ LambdaSignature {
+ type_signature,
+ volatility,
+ }
+ }
+
+ /// User-defined coercion rules for the function.
+ pub fn user_defined(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::UserDefined,
+ volatility,
+ }
+ }
+
+ /// An arbitrary number of lambdas or arguments of any type.
+ pub fn variadic_any(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::VariadicAny,
+ volatility,
+ }
+ }
+
+ /// A specified number of arguments of any type
+ pub fn any(arg_count: usize, volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::Any(arg_count),
+ volatility,
+ }
+ }
+}
+
+impl PartialEq for dyn LambdaUDF {
+ fn eq(&self, other: &Self) -> bool {
+ self.dyn_eq(other.as_any())
+ }
+}
+
+impl PartialOrd for dyn LambdaUDF {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ let mut cmp = self.name().cmp(other.name());
+ if cmp == Ordering::Equal {
+ cmp = self.signature().partial_cmp(other.signature())?;
+ }
+ if cmp == Ordering::Equal {
+ cmp = self.aliases().partial_cmp(other.aliases())?;
+ }
+ // Contract for PartialOrd and PartialEq consistency requires that
+ // a == b if and only if partial_cmp(a, b) == Some(Equal).
+ if cmp == Ordering::Equal && self != other {
+ // Functions may have other properties besides name and signature
+ // that differentiate two instances (e.g. type, or arbitrary parameters).
+ // We cannot return Some(Equal) in such case.
+ return None;
+ }
+ debug_assert!(
+ cmp == Ordering::Equal || self != other,
+ "Detected incorrect implementation of PartialEq when comparing functions: '{}' and '{}'. \
+ The functions compare as equal, but they are not equal based on general properties that \
+ the PartialOrd implementation observes,",
+ self.name(),
+ other.name()
+ );
+ Some(cmp)
+ }
+}
+
+impl Eq for dyn LambdaUDF {}
+
+impl Hash for dyn LambdaUDF {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.dyn_hash(state)
+ }
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+#[derive(Debug, Clone)]
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments and lambdas to the function
+ pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>,
+ /// Field associated with each arg, if it exists
+ /// For lambdas, it will be the field of the result of
+ /// the lambda if evaluated with the parameters
+ /// returned from [`LambdaUDF::lambdas_parameters`]
+ pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned
+ /// (from `return_field_from_args`) when creating the
+ /// physical expression from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+impl LambdaFunctionArgs {
+ /// The return type of the function. See [`Self::return_field`] for more
+ /// details.
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+}
+
+/// A lambda argument to a LambdaFunction
+#[derive(Clone, Debug)]
+pub struct LambdaArgument {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ body: Arc<dyn PhysicalExpr>,
+}
+
+impl LambdaArgument {
+ pub fn new(params: Vec<FieldRef>, body: Arc<dyn PhysicalExpr>) -> Self {
+ Self { params, body }
+ }
+
+ /// Evaluate this lambda
+ /// `args` should evaluate to the value of each parameter
+ /// of the correspondent lambda returned in [LambdaUDF::lambdas_parameters].
+ pub fn evaluate(
+ &self,
+ args: &[&dyn Fn() -> Result<ArrayRef>],
+ ) -> Result<ColumnarValue> {
+ let columns = args
+ .iter()
+ .take(self.params.len())
+ .map(|arg| arg())
+ .collect::<Result<_>>()?;
+
+ let schema = Arc::new(Schema::new(self.params.clone()));
+
+ let batch = RecordBatch::try_new(schema, columns)?;
+
+ self.body.evaluate(&batch)
+ }
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+#[derive(Clone, Debug)]
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field of the result of the
+ /// lambda if evaluated with the parameters returned from [`LambdaUDF::lambdas_parameters`]
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ /// ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)),
+ /// ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false))
+ /// ]`
+ pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `array_transform([1], v -> v == 5)`
+ /// this field will be `[Some(ScalarValue::List(...), None]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// An argument to a lambda function
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum ValueOrLambda<V, L> {
+ /// A value with associated data
+ Value(V),
+ /// A lambda with associated data
+ Lambda(L),
+}
+
+/// Trait for implementing user defined lambda functions.
+///
+/// This trait exposes the full API for implementing user defined functions and
+/// can be used to implement any function.
+///
+/// See [`array_transform.rs`] for a commented complete implementation
+///
+/// [`array_transform.rs`]: https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/array_transform.rs
+pub trait LambdaUDF: Debug + DynEq + DynHash + Send + Sync {
+ /// Returns this object as an [`Any`] trait object
+ fn as_any(&self) -> &dyn Any;
+
+ /// Returns this function's name
+ fn name(&self) -> &str;
+
+ /// Returns any aliases (alternate names) for this function.
+ ///
+ /// Aliases can be used to invoke the same function using different names.
+ /// For example in some databases `now()` and `current_timestamp()` are
+ /// aliases for the same function. This behavior can be obtained by
+ /// returning `current_timestamp` as an alias for the `now` function.
+ ///
+ /// Note: `aliases` should only include names other than [`Self::name`].
+ /// Defaults to `[]` (no aliases)
+ fn aliases(&self) -> &[String] {
+ &[]
+ }
+
+ /// Returns the name of the column this expression would create
+ ///
+ /// See [`Expr::schema_name`] for details
+ fn schema_name(&self, args: &[Expr]) -> Result<String> {
+ Ok(format!(
+ "{}({})",
+ self.name(),
+ schema_name_from_exprs_comma_separated_without_space(args)?
+ ))
+ }
+
+ /// Returns a [`LambdaSignature`] describing the argument types for which this
+ /// function has an implementation, and the function's [`Volatility`].
+ ///
+ /// See [`LambdaSignature`] for more details on argument type handling
+ /// and [`Self::return_field_from_args`] for computing the return type.
+ ///
+ /// [`Volatility`]: datafusion_expr_common::signature::Volatility
+ fn signature(&self) -> &LambdaSignature;
+
+ /// Return the field of all the parameters supported by all the supported lambdas of this function
+ /// based on the field of the value arguments. If a lambda support multiple parameters, or if multiple
+ /// lambdas are supported and some are optional, all should be returned,
+ /// regardless of whether they are used on a particular invocation
+ ///
+ /// Tip: If you have a [`LambdaFunction`] invocation, you can call the helper
+ /// [`LambdaFunction::lambdas_parameters`] instead of this method directly
+ ///
+ /// [`LambdaFunction`]: crate::expr::LambdaFunction
+ /// [`LambdaFunction::lambdas_parameters`]: crate::expr::LambdaFunction::lambdas_parameters
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2.0, 8.0], v -> v > 4.0)`
+ ///
+ /// ```ignore
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// Arc::new(Field::new("", DataType::new_list(DataType::Float32, false))), // the Field of the literal `[2, 8]`
+ /// ])?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// // the lambda supported parameters, regardless of how many are actually used
+ /// vec![
+ /// // the value being transformed
+ /// Field::new("", DataType::Float32, false),
+ /// // the 1-based index being transformed, not used on the example above,
+ /// //but implementations doesn't need to care about it
+ /// Field::new("", DataType::Int32, false),
+ /// ]
+ /// ]
+ /// )
+ /// ```
+ ///
+ /// The implementation can assume that some other part of the code has coerced
+ /// the actual argument types to match [`Self::signature`].
+ fn lambdas_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>>;
Review Comment:
nit: The double plural is a bit odd to read. Doesn't seem to match the style used elsewhere in the codebase for functions returning a `Vec<Vec<>>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3005947190
##########
datafusion/expr/src/udlf.rs:
##########
@@ -0,0 +1,548 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`]: Lambda User Defined Functions
+
+use crate::expr::schema_name_from_exprs_comma_separated_without_space;
+use crate::{ColumnarValue, Documentation, Expr};
+use arrow::array::{ArrayRef, RecordBatch};
+use arrow::datatypes::{DataType, Field, FieldRef, Schema};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{Result, ScalarValue, not_impl_err};
+use datafusion_expr_common::dyn_eq::{DynEq, DynHash};
+use datafusion_expr_common::signature::Volatility;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::any::Any;
+use std::cmp::Ordering;
+use std::fmt::Debug;
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+/// The types of arguments for which a function has implementations.
+///
+/// [`LambdaTypeSignature`] **DOES NOT** define the types that a user query could call the
+/// function with. DataFusion will automatically coerce (cast) argument types to
+/// one of the supported function signatures, if possible.
+///
+/// # Overview
+/// Functions typically provide implementations for a small number of different
+/// argument [`DataType`]s, rather than all possible combinations. If a user
+/// calls a function with arguments that do not match any of the declared types,
+/// DataFusion will attempt to automatically coerce (add casts to) function
+/// arguments so they match the [`LambdaTypeSignature`]. See the [`type_coercion`] module
+/// for more details
+///
+/// [`type_coercion`]: crate::type_coercion
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub enum LambdaTypeSignature {
+ /// The acceptable signature and coercions rules are special for this
+ /// function.
+ ///
+ /// If this signature is specified,
+ /// DataFusion will call [`LambdaUDF::coerce_value_types`] to prepare argument types.
+ UserDefined,
+ /// One or more lambdas or arguments with arbitrary types
+ VariadicAny,
+ /// The specified number of lambdas or arguments with arbitrary types.
+ Any(usize),
+}
+
+/// Provides information necessary for calling a lambda function.
+///
+/// - [`LambdaTypeSignature`] defines the argument types that a function has implementations
+/// for.
+///
+/// - [`Volatility`] defines how the output of the function changes with the input.
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub struct LambdaSignature {
+ /// The data types that the function accepts. See [LambdaTypeSignature] for more information.
+ pub type_signature: LambdaTypeSignature,
+ /// The volatility of the function. See [Volatility] for more information.
+ pub volatility: Volatility,
+}
+
+impl LambdaSignature {
+ /// Creates a new `LambdaSignature` from a given type signature and volatility.
+ pub fn new(type_signature: LambdaTypeSignature, volatility: Volatility) -> Self {
+ LambdaSignature {
+ type_signature,
+ volatility,
+ }
+ }
+
+ /// User-defined coercion rules for the function.
+ pub fn user_defined(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::UserDefined,
+ volatility,
+ }
+ }
+
+ /// An arbitrary number of lambdas or arguments of any type.
+ pub fn variadic_any(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::VariadicAny,
+ volatility,
+ }
+ }
+
+ /// A specified number of arguments of any type
+ pub fn any(arg_count: usize, volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::Any(arg_count),
+ volatility,
+ }
+ }
+}
+
+impl PartialEq for dyn LambdaUDF {
+ fn eq(&self, other: &Self) -> bool {
+ self.dyn_eq(other.as_any())
+ }
+}
+
+impl PartialOrd for dyn LambdaUDF {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ let mut cmp = self.name().cmp(other.name());
+ if cmp == Ordering::Equal {
+ cmp = self.signature().partial_cmp(other.signature())?;
+ }
+ if cmp == Ordering::Equal {
+ cmp = self.aliases().partial_cmp(other.aliases())?;
+ }
+ // Contract for PartialOrd and PartialEq consistency requires that
+ // a == b if and only if partial_cmp(a, b) == Some(Equal).
+ if cmp == Ordering::Equal && self != other {
+ // Functions may have other properties besides name and signature
+ // that differentiate two instances (e.g. type, or arbitrary parameters).
+ // We cannot return Some(Equal) in such case.
+ return None;
+ }
+ debug_assert!(
+ cmp == Ordering::Equal || self != other,
+ "Detected incorrect implementation of PartialEq when comparing functions: '{}' and '{}'. \
+ The functions compare as equal, but they are not equal based on general properties that \
+ the PartialOrd implementation observes,",
+ self.name(),
+ other.name()
+ );
+ Some(cmp)
+ }
+}
+
+impl Eq for dyn LambdaUDF {}
+
+impl Hash for dyn LambdaUDF {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.dyn_hash(state)
+ }
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+#[derive(Debug, Clone)]
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments and lambdas to the function
+ pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>,
+ /// Field associated with each arg, if it exists
+ /// For lambdas, it will be the field of the result of
+ /// the lambda if evaluated with the parameters
+ /// returned from [`LambdaUDF::lambdas_parameters`]
+ pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned
+ /// (from `return_field_from_args`) when creating the
+ /// physical expression from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+impl LambdaFunctionArgs {
+ /// The return type of the function. See [`Self::return_field`] for more
+ /// details.
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+}
+
+/// A lambda argument to a LambdaFunction
+#[derive(Clone, Debug)]
+pub struct LambdaArgument {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ body: Arc<dyn PhysicalExpr>,
+}
+
+impl LambdaArgument {
+ pub fn new(params: Vec<FieldRef>, body: Arc<dyn PhysicalExpr>) -> Self {
+ Self { params, body }
+ }
+
+ /// Evaluate this lambda
+ /// `args` should evaluate to the value of each parameter
+ /// of the correspondent lambda returned in [LambdaUDF::lambdas_parameters].
+ pub fn evaluate(
+ &self,
+ args: &[&dyn Fn() -> Result<ArrayRef>],
+ ) -> Result<ColumnarValue> {
+ let columns = args
+ .iter()
+ .take(self.params.len())
+ .map(|arg| arg())
+ .collect::<Result<_>>()?;
+
+ let schema = Arc::new(Schema::new(self.params.clone()));
+
+ let batch = RecordBatch::try_new(schema, columns)?;
+
+ self.body.evaluate(&batch)
+ }
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+#[derive(Clone, Debug)]
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field of the result of the
+ /// lambda if evaluated with the parameters returned from [`LambdaUDF::lambdas_parameters`]
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ /// ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)),
+ /// ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false))
+ /// ]`
+ pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `array_transform([1], v -> v == 5)`
+ /// this field will be `[Some(ScalarValue::List(...), None]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// An argument to a lambda function
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum ValueOrLambda<V, L> {
+ /// A value with associated data
+ Value(V),
+ /// A lambda with associated data
+ Lambda(L),
+}
+
+/// Trait for implementing user defined lambda functions.
+///
+/// This trait exposes the full API for implementing user defined functions and
+/// can be used to implement any function.
+///
+/// See [`array_transform.rs`] for a commented complete implementation
+///
+/// [`array_transform.rs`]: https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/array_transform.rs
+pub trait LambdaUDF: Debug + DynEq + DynHash + Send + Sync {
+ /// Returns this object as an [`Any`] trait object
+ fn as_any(&self) -> &dyn Any;
+
+ /// Returns this function's name
+ fn name(&self) -> &str;
+
+ /// Returns any aliases (alternate names) for this function.
+ ///
+ /// Aliases can be used to invoke the same function using different names.
+ /// For example in some databases `now()` and `current_timestamp()` are
+ /// aliases for the same function. This behavior can be obtained by
+ /// returning `current_timestamp` as an alias for the `now` function.
+ ///
+ /// Note: `aliases` should only include names other than [`Self::name`].
+ /// Defaults to `[]` (no aliases)
+ fn aliases(&self) -> &[String] {
+ &[]
+ }
+
+ /// Returns the name of the column this expression would create
+ ///
+ /// See [`Expr::schema_name`] for details
+ fn schema_name(&self, args: &[Expr]) -> Result<String> {
+ Ok(format!(
+ "{}({})",
+ self.name(),
+ schema_name_from_exprs_comma_separated_without_space(args)?
+ ))
+ }
+
+ /// Returns a [`LambdaSignature`] describing the argument types for which this
+ /// function has an implementation, and the function's [`Volatility`].
+ ///
+ /// See [`LambdaSignature`] for more details on argument type handling
+ /// and [`Self::return_field_from_args`] for computing the return type.
+ ///
+ /// [`Volatility`]: datafusion_expr_common::signature::Volatility
+ fn signature(&self) -> &LambdaSignature;
+
+ /// Return the field of all the parameters supported by all the supported lambdas of this function
+ /// based on the field of the value arguments. If a lambda support multiple parameters, or if multiple
+ /// lambdas are supported and some are optional, all should be returned,
+ /// regardless of whether they are used on a particular invocation
+ ///
+ /// Tip: If you have a [`LambdaFunction`] invocation, you can call the helper
+ /// [`LambdaFunction::lambdas_parameters`] instead of this method directly
+ ///
+ /// [`LambdaFunction`]: crate::expr::LambdaFunction
+ /// [`LambdaFunction::lambdas_parameters`]: crate::expr::LambdaFunction::lambdas_parameters
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2.0, 8.0], v -> v > 4.0)`
+ ///
+ /// ```ignore
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// Arc::new(Field::new("", DataType::new_list(DataType::Float32, false))), // the Field of the literal `[2, 8]`
+ /// ])?;
+ ///
+ /// assert_eq!(
+ /// lambdas_parameters,
+ /// vec![
+ /// // the lambda supported parameters, regardless of how many are actually used
+ /// vec![
+ /// // the value being transformed
+ /// Field::new("", DataType::Float32, false),
+ /// // the 1-based index being transformed, not used on the example above,
+ /// //but implementations doesn't need to care about it
+ /// Field::new("", DataType::Int32, false),
+ /// ]
+ /// ]
+ /// )
+ /// ```
+ ///
+ /// The implementation can assume that some other part of the code has coerced
+ /// the actual argument types to match [`Self::signature`].
+ fn lambdas_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>>;
Review Comment:
Done at https://github.com/apache/datafusion/pull/18921/changes/69c44fc334150a241f8bf53792fc3d9377883497 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3005982227
##########
datafusion/expr/src/expr.rs:
##########
@@ -404,6 +406,104 @@ pub enum Expr {
OuterReferenceColumn(FieldRef, Column),
/// Unnest expression
Unnest(Unnest),
+ /// Call a lambda function with a set of arguments.
+ ///
+ /// For example, `array_transform([1,2,3], v -> v+1)` would be equivalent to:
+ ///
+ /// ```text
+ /// LambdaFunction(array_transform)
+ /// ├── args[0]: Literal([1,2,3])
+ /// └── args[1]: Lambda
+ /// ├── params: ["v"]
+ /// └── body: BinaryExpr(+)
+ /// ├── LambdaVariable("v")
+ /// └── Literal(1)
+ /// ```
+ LambdaFunction(LambdaFunction),
Review Comment:
Hmm, definitely should be higher-order function, yes https://github.com/apache/datafusion/pull/18921/changes/66839d3a76c8b6a3029c97b7472e2534ce543bd3, thank you. Other symbols got renamed as well like `LambdaUDF` to `HigherOrderUDF`, `LambdaSignature` to `HigherOrderSignature`, etc
For now it supports only anonymous inline functions, yes, but we can add support for it too. Maybe with a [FunctionRewrite](https://docs.rs/datafusion/latest/datafusion/logical_expr/expr_rewriter/trait.FunctionRewrite.html) that rewrites the scalar UDF name into an anonynomous inline function? (I believe it can be done out-of-tree as well)
I imagine that trying to natively support names would hide the scalar udf call from tree traversals that may be necessary, like type coercion, and also somehow require the function instance access to the specific scalar udf it got called, like reinstancing the function with the udf, or providing the scalar functions map to `HigherOrderFunction` methods `invoke_with_args` and `return_field_from_args`, which wouldn't be as trivial as the `FunctionRewrite` approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3010054120
##########
datafusion/execution/src/task.rs:
##########
Review Comment:
Yes, https://github.com/apache/datafusion/pull/18921/changes/72558201bbc00641d2a3b426417e0eb42cbd7169, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976118931
##########
datafusion/expr/src/udlf.rs:
##########
@@ -0,0 +1,550 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`]: Lambda User Defined Functions
+
+use crate::expr::schema_name_from_exprs_comma_separated_without_space;
+use crate::{ColumnarValue, Documentation, Expr};
+use arrow::array::{ArrayRef, RecordBatch};
+use arrow::datatypes::{DataType, Field, FieldRef, Schema};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{Result, ScalarValue, not_impl_err};
+use datafusion_expr_common::dyn_eq::{DynEq, DynHash};
+use datafusion_expr_common::signature::Volatility;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use std::any::Any;
+use std::cmp::Ordering;
+use std::fmt::Debug;
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+/// The types of arguments for which a function has implementations.
+///
+/// [`LambdaTypeSignature`] **DOES NOT** define the types that a user query could call the
+/// function with. DataFusion will automatically coerce (cast) argument types to
+/// one of the supported function signatures, if possible.
+///
+/// # Overview
+/// Functions typically provide implementations for a small number of different
+/// argument [`DataType`]s, rather than all possible combinations. If a user
+/// calls a function with arguments that do not match any of the declared types,
+/// DataFusion will attempt to automatically coerce (add casts to) function
+/// arguments so they match the [`LambdaTypeSignature`]. See the [`type_coercion`] module
+/// for more details
+///
+/// [`type_coercion`]: crate::type_coercion
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub enum LambdaTypeSignature {
+ /// The acceptable signature and coercions rules are special for this
+ /// function.
+ ///
+ /// If this signature is specified,
+ /// DataFusion will call [`LambdaUDF::coerce_value_types`] to prepare argument types.
+ UserDefined,
+ /// One or more lambdas or arguments with arbitrary types
+ VariadicAny,
+ /// The specified number of lambdas or arguments with arbitrary types.
+ Any(usize),
+}
+
+/// Provides information necessary for calling a lambda function.
+///
+/// - [`LambdaTypeSignature`] defines the argument types that a function has implementations
+/// for.
+///
+/// - [`Volatility`] defines how the output of the function changes with the input.
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
+pub struct LambdaSignature {
+ /// The data types that the function accepts. See [LambdaTypeSignature] for more information.
+ pub type_signature: LambdaTypeSignature,
+ /// The volatility of the function. See [Volatility] for more information.
+ pub volatility: Volatility,
+}
+
+impl LambdaSignature {
+ /// Creates a new `LambdaSignature` from a given type signature and volatility.
+ pub fn new(type_signature: LambdaTypeSignature, volatility: Volatility) -> Self {
+ LambdaSignature {
+ type_signature,
+ volatility,
+ }
+ }
+
+ /// User-defined coercion rules for the function.
+ pub fn user_defined(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::UserDefined,
+ volatility,
+ }
+ }
+
+ /// An arbitrary number of lambdas or arguments of any type.
+ pub fn variadic_any(volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::VariadicAny,
+ volatility,
+ }
+ }
+
+ /// A specified number of arguments of any type
+ pub fn any(arg_count: usize, volatility: Volatility) -> Self {
+ Self {
+ type_signature: LambdaTypeSignature::Any(arg_count),
+ volatility,
+ }
+ }
+}
+
+impl PartialEq for dyn LambdaUDF {
+ fn eq(&self, other: &Self) -> bool {
+ self.dyn_eq(other.as_any())
+ }
+}
+
+impl PartialOrd for dyn LambdaUDF {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ let mut cmp = self.name().cmp(other.name());
+ if cmp == Ordering::Equal {
+ cmp = self.signature().partial_cmp(other.signature())?;
+ }
+ if cmp == Ordering::Equal {
+ cmp = self.aliases().partial_cmp(other.aliases())?;
+ }
+ // Contract for PartialOrd and PartialEq consistency requires that
+ // a == b if and only if partial_cmp(a, b) == Some(Equal).
+ if cmp == Ordering::Equal && self != other {
+ // Functions may have other properties besides name and signature
+ // that differentiate two instances (e.g. type, or arbitrary parameters).
+ // We cannot return Some(Equal) in such case.
+ return None;
+ }
+ debug_assert!(
+ cmp == Ordering::Equal || self != other,
+ "Detected incorrect implementation of PartialEq when comparing functions: '{}' and '{}'. \
+ The functions compare as equal, but they are not equal based on general properties that \
+ the PartialOrd implementation observes,",
+ self.name(),
+ other.name()
+ );
+ Some(cmp)
+ }
+}
+
+impl Eq for dyn LambdaUDF {}
+
+impl Hash for dyn LambdaUDF {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.dyn_hash(state)
+ }
+}
+
+/// Arguments passed to [`LambdaUDF::invoke_with_args`] when invoking a
+/// lambda function.
+#[derive(Debug, Clone)]
+pub struct LambdaFunctionArgs {
+ /// The evaluated arguments and lambdas to the function
+ pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>,
+ /// Field associated with each arg, if it exists
+ /// For lambdas, it will be the field of the result of
+ /// the lambda if evaluated with the parameters
+ /// returned from [`LambdaUDF::lambdas_parameters`]
+ pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>,
+ /// The number of rows in record batch being evaluated
+ pub number_rows: usize,
+ /// The return field of the lambda function returned
+ /// (from `return_field_from_args`) when creating the
+ /// physical expression from the logical expression
+ pub return_field: FieldRef,
+ /// The config options at execution time
+ pub config_options: Arc<ConfigOptions>,
+}
+
+impl LambdaFunctionArgs {
+ /// The return type of the function. See [`Self::return_field`] for more
+ /// details.
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+}
+
+/// A lambda argument to a LambdaFunction
+#[derive(Clone, Debug)]
+pub struct LambdaArgument {
+ /// The parameters defined in this lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be `vec![Field::new("v", DataType::Int32, true)]`
+ params: Vec<FieldRef>,
+ /// The body of the lambda
+ ///
+ /// For example, for `array_transform([2], v -> -v)`,
+ /// this will be the physical expression of `-v`
+ body: Arc<dyn PhysicalExpr>,
+}
+
+impl LambdaArgument {
+ pub fn new(params: Vec<FieldRef>, body: Arc<dyn PhysicalExpr>) -> Self {
+ Self { params, body }
+ }
+
+ /// Evaluate this lambda
+ /// `args` should evaluate to the value of each parameter
+ /// of the correspondent lambda returned in [LambdaUDF::lambdas_parameters].
+ pub fn evaluate(
+ &self,
+ args: &[&dyn Fn() -> Result<ArrayRef>],
+ ) -> Result<ColumnarValue> {
+ let columns = args
+ .iter()
+ .take(self.params.len())
+ .map(|arg| arg())
+ .collect::<Result<_>>()?;
+
+ let schema = Arc::new(Schema::new(self.params.clone()));
+
+ let batch = RecordBatch::try_new(schema, columns)?;
+
+ self.body.evaluate(&batch)
+ }
+}
+
+/// Information about arguments passed to the function
+///
+/// This structure contains metadata about how the function was called
+/// such as the type of the arguments, any scalar arguments and if the
+/// arguments can (ever) be null
+///
+/// See [`LambdaUDF::return_field_from_args`] for more information
+#[derive(Clone, Debug)]
+pub struct LambdaReturnFieldArgs<'a> {
+ /// The data types of the arguments to the function
+ ///
+ /// If argument `i` to the function is a lambda, it will be the field of the result of the
+ /// lambda if evaluated with the parameters returned from [`LambdaUDF::lambdas_parameters`]
+ ///
+ /// For example, with `array_transform([1], v -> v == 5)`
+ /// this field will be `[
+ /// ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)),
+ /// ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false))
+ /// ]`
+ pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>],
+ /// Is argument `i` to the function a scalar (constant)?
+ ///
+ /// If the argument `i` is not a scalar, it will be None
+ ///
+ /// For example, if a function is called like `array_transform([1], v -> v == 5)`
+ /// this field will be `[Some(ScalarValue::List(...), None]`
+ pub scalar_arguments: &'a [Option<&'a ScalarValue>],
+}
+
+/// An argument to a lambda function
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum ValueOrLambda<V, L> {
+ /// A value with associated data
+ Value(V),
+ /// A lambda with associated data
+ Lambda(L),
+}
+
+/// Trait for implementing user defined lambda functions.
+///
+/// This trait exposes the full API for implementing user defined functions and
+/// can be used to implement any function.
+///
+/// See [`array_transform.rs`] for a commented complete implementation
+///
+/// [`array_transform.rs`]: https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/array_transform.rs
+pub trait LambdaUDF: Debug + DynEq + DynHash + Send + Sync {
+ /// Returns this object as an [`Any`] trait object
+ fn as_any(&self) -> &dyn Any;
+
+ /// Returns this function's name
+ fn name(&self) -> &str;
+
+ /// Returns any aliases (alternate names) for this function.
+ ///
+ /// Aliases can be used to invoke the same function using different names.
+ /// For example in some databases `now()` and `current_timestamp()` are
+ /// aliases for the same function. This behavior can be obtained by
+ /// returning `current_timestamp` as an alias for the `now` function.
+ ///
+ /// Note: `aliases` should only include names other than [`Self::name`].
+ /// Defaults to `[]` (no aliases)
+ fn aliases(&self) -> &[String] {
+ &[]
+ }
+
+ /// Returns the name of the column this expression would create
+ ///
+ /// See [`Expr::schema_name`] for details
+ fn schema_name(&self, args: &[Expr]) -> Result<String> {
+ Ok(format!(
+ "{}({})",
+ self.name(),
+ schema_name_from_exprs_comma_separated_without_space(args)?
+ ))
+ }
+
+ /// Returns a [`LambdaSignature`] describing the argument types for which this
+ /// function has an implementation, and the function's [`Volatility`].
+ ///
+ /// See [`LambdaSignature`] for more details on argument type handling
+ /// and [`Self::return_field_from_args`] for computing the return type.
+ ///
+ /// [`Volatility`]: datafusion_expr_common::signature::Volatility
+ fn signature(&self) -> &LambdaSignature;
+
+ /// Returns a list of the same size as args where each value is the logic below applied to value at the correspondent position in args:
+ ///
+ /// If it's a value, return None
+ /// If it's a lambda, return the list of all parameters that that lambda supports
+ ///
+ /// Example for array_transform:
+ ///
+ /// `array_transform([2.0, 8.0], v -> v > 4.0)`
+ ///
+ /// ```ignore
+ /// let lambdas_parameters = array_transform.lambdas_parameters(&[
+ /// ValueOrLambdaParameter::Value(Field::new("", DataType::new_list(DataType::Float32, false)))]), // the Field of the literal `[2, 8]`
+ /// ValueOrLambdaParameter::Lambda, // A lambda
Review Comment:
I will apply your suggestions regarding lambdas_parameters later and this will become a vec of Field's, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2976002894
##########
datafusion/expr/src/type_coercion/functions.rs:
##########
@@ -148,6 +151,74 @@ pub fn fields_with_udf<F: UDFCoercionExt>(
.collect())
}
+/// Performs type coercion for lambda function arguments.
+///
+/// For value arguments, returns the field to which each
+/// argument must be coerced to match `signature`.
+/// For lambda arguments, returns a clone of the associated data
+///
+/// For more details on coercion in general, please see the
+/// [`type_coercion`](crate::type_coercion) module.
+pub fn value_fields_with_lambda_udf<L: Clone>(
+ current_fields: &[ValueOrLambda<FieldRef, L>],
+ func: &dyn LambdaUDF,
+) -> Result<Vec<ValueOrLambda<FieldRef, L>>> {
+ match func.signature().type_signature {
+ LambdaTypeSignature::UserDefined => {
+ let arg_types = current_fields
+ .iter()
+ .filter_map(|p| match p {
+ ValueOrLambda::Value(field) => Some(field.data_type().clone()),
+ ValueOrLambda::Lambda(_) => None,
+ })
+ .collect::<Vec<_>>();
+
+ let coerced_types = func.coerce_value_types(&arg_types)?;
+
+ if coerced_types.len() != arg_types.len() {
+ return plan_err!(
+ "{} coerce_value_types should have returned {} items but returned {}",
+ func.name(),
+ arg_types.len(),
+ coerced_types.len()
+ );
+ }
+
+ let mut coerced_types = coerced_types.into_iter();
+
+ Ok(current_fields
Review Comment:
Sure https://github.com/apache/datafusion/pull/18921/changes/a241a5148830ae7e374e3add89d206d8f48a08b0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2996310278
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ let (list, _lambda) = value_lambda_pair(self.name(), args)?;
+
+ let field = match list.data_type() {
+ DataType::List(field) => field,
+ DataType::LargeList(field) => field,
+ DataType::FixedSizeList(field, _) => field,
Review Comment:
ah I completely missed this was already handled by type coercion above, you are right, for a first version it's probably fine as it is now. We can implement the map_transform after as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r2996310278
##########
datafusion/functions-nested/src/array_transform.rs:
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`LambdaUDF`] definitions for array_transform function.
+
+use arrow::{
+ array::{Array, ArrayRef, AsArray, FixedSizeListArray, LargeListArray, ListArray},
+ datatypes::{DataType, Field, FieldRef},
+};
+use datafusion_common::{
+ Result, exec_err, plan_err,
+ utils::{list_values, take_function_args},
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, LambdaFunctionArgs, LambdaReturnFieldArgs,
+ LambdaSignature, LambdaUDF, ValueOrLambda, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::{any::Any, fmt::Debug, sync::Arc};
+
+make_udlf_expr_and_func!(
+ ArrayTransform,
+ array_transform,
+ array lambda,
+ "transforms the values of a array",
+ array_transform_udlf
+);
+
+#[user_doc(
+ doc_section(label = "Array Functions"),
+ description = "transforms the values of a array",
+ syntax_example = "array_transform(array, x -> x*2)",
+ sql_example = r#"```sql
+> select array_transform([1, 2, 3, 4, 5], x -> x*2);
++-------------------------------------------+
+| array_transform([1, 2, 3, 4, 5], x -> x*2) |
++-------------------------------------------+
+| [2, 4, 6, 8, 10] |
++-------------------------------------------+
+```"#,
+ argument(
+ name = "array",
+ description = "Array expression. Can be a constant, column, or function, and any combination of array operators."
+ ),
+ argument(name = "lambda", description = "Lambda")
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ArrayTransform {
+ signature: LambdaSignature,
+ aliases: Vec<String>,
+}
+
+impl Default for ArrayTransform {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ArrayTransform {
+ pub fn new() -> Self {
+ Self {
+ signature: LambdaSignature::user_defined(Volatility::Immutable),
+ aliases: vec![String::from("list_transform")],
+ }
+ }
+}
+
+impl LambdaUDF for ArrayTransform {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "array_transform"
+ }
+
+ fn aliases(&self) -> &[String] {
+ &self.aliases
+ }
+
+ fn signature(&self) -> &LambdaSignature {
+ &self.signature
+ }
+
+ fn coerce_value_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ let list = if arg_types.len() == 1 {
+ &arg_types[0]
+ } else {
+ return plan_err!(
+ "{} function requires 1 value arguments, got {}",
+ self.name(),
+ arg_types.len()
+ );
+ };
+
+ let coerced = match list {
+ DataType::List(_)
+ | DataType::LargeList(_)
+ | DataType::FixedSizeList(_, _) => list.clone(),
+ DataType::ListView(field) => DataType::List(Arc::clone(field)),
+ DataType::LargeListView(field) => DataType::LargeList(Arc::clone(field)),
+ _ => {
+ return plan_err!(
+ "{} expected a list as first argument, got {}",
+ self.name(),
+ list
+ );
+ }
+ };
+
+ Ok(vec![coerced])
+ }
+
+ fn lambdas_parameters(
+ &self,
+ args: &[ValueOrLambda<FieldRef, ()>],
+ ) -> Result<Vec<Option<Vec<Field>>>> {
+ let (list, _lambda) = value_lambda_pair(self.name(), args)?;
+
+ let field = match list.data_type() {
+ DataType::List(field) => field,
+ DataType::LargeList(field) => field,
+ DataType::FixedSizeList(field, _) => field,
Review Comment:
ah I completely missed this was already handled by type coercion above, you are right, for a first version it's probably fine as it is now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "linhr (via GitHub)" <gi...@apache.org>.
linhr commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3668603319
This is an exciting initiative!
I skimmed through the PR, and I have some thoughts regarding how this can fit into DataFusion's existing setup. I wonder if we can have `Expr::LambdaFunction(LambdaFunction)`, similar to `Expr::ScalarFunction(ScalarFunction)`? Here are my reasoning:
1. `LambdaFunction` can represent a "resolved" lambda function call in the logical plan. In contrast, `Expr::Lambda` and `Expr::LambdaColumn` are only fragments whose data types etc. are not well-defined which may be challenging to work with in other parts of the library (e.g. `ExprSchemable`).
2. I feel `ScalarUDF` may not fit cleanly for lambda functions, so we can then have a separate abstraction (e.g. `LambdaUDF`) inside `Expr::LambdaFunction`. The `ArrayTransform` example shows that we would have to use `ScalarFunctionArgs` which was not originally designed for the lambda use case. (IMO when the function is actually "invoked" with Arrow arrays, the lambda parameter should have been resolved and removed from the argument list.)
3. Once we have a self-contained logical expression, the entire query analysis flow might become easier to reason about (from `SqlToRel` to `Expr::LambdaFunction` to a dedicated `PhysicalExpr` and the corresponding physical planner).
This is just my rough thoughts. Happy to discuss!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [RFC] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3651816409
Thanks @shehabgamin. I analyzed the Spark implementation one last time, and realized I could have done similarly since the beginning... I've already tested it locally and pretend to push soon, the diff reduced to ~2K LOC, mostly in new, self contained code with small changes in existing code and without requiring changes in downstream code. I hope you haven't put time reviewing it yet because I believe the spark approach is much better and also easier/faster to review
Basically, for planning, we lazily set the Field in the lambda column itself, instead of injecting it into a DFSchema
```rust
//logical
struct LambdaColumn {
name: String,
field: Option<FieldRef>,
}
```
And for evaluation, again, we lazily set the value of a lambda column instead of injecting it into a RecordBatch
```rust
//physical
struct LambdaColumn {
name: String,
field: FieldRef,
value: Option<ColumnarValue>,
}
```
I think I initially ruled that out because other expressions doesn't contain it's own Field, and this have been discussed before in #12604 *, and didn't went forward, which the difference that the other expressions Field's naturally exist in the plan schema, which *now* I believe justifies this difference.
And most importantly, because I had no idea of how much work and downstream churn injecting the lambdas parameters into the Schema/RecordBatch would cause :facepalm:
I will push soon and update the PR description, thanks again!
*This is being discussed again in #18845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4236994808
@gstvg I reviewed the pr, once the comments are resolved and the merge conflicts are resolved as well I will merge this if no no one have objection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-4236985563
To view the entire breaking change blast radius this would have been helpful:
- #21406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3074566849
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
Review Comment:
https://github.com/apache/datafusion/pull/18921/changes/954a3603a003fe02a7f4629df0825c0a9a895a4f thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add lambda support and array_transform udf [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#discussion_r3081123480
##########
datafusion/physical-expr/src/higher_order_function.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Declaration of built-in (higher order) functions.
+//! This module contains built-in functions' enumeration and metadata.
+//!
+//! Generally, a function has:
+//! * a signature
+//! * a return type, that is a function of the incoming argument's types
+//! * the computation, that must accept each valid signature
+//!
+//! * Signature: see `Signature`
+//! * Return type: a function `(arg_types) -> return_type`. E.g. for array_transform, ([[f32]], v -> v*2) -> [f32], ([[f32]], v -> v > 3.0) -> [bool].
+//!
+//! This module also has a set of coercion rules to improve user experience: if an argument i32 is passed
+//! to a function that supports f64, it is coerced to f64.
+
+use std::any::Any;
+use std::fmt::{self, Debug, Formatter};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+use crate::expressions::{LambdaExpr, Literal};
+
+use arrow::array::{Array, RecordBatch};
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use datafusion_common::config::{ConfigEntry, ConfigOptions};
+use datafusion_common::utils::remove_list_null_values;
+use datafusion_common::{
+ Result, ScalarValue, exec_err, internal_datafusion_err, internal_err,
+};
+use datafusion_expr::type_coercion::functions::value_fields_with_higher_order_udf;
+use datafusion_expr::{
+ ColumnarValue, HigherOrderFunctionArgs, HigherOrderReturnFieldArgs, HigherOrderUDF,
+ LambdaArgument, ValueOrLambda, Volatility, expr_vec_fmt,
+};
+
+/// Physical expression of a higher order function
+pub struct HigherOrderFunctionExpr {
+ fun: Arc<dyn HigherOrderUDF>,
+ name: String,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+}
+
+impl Debug for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ f.debug_struct("HigherOrderFunctionExpr")
+ .field("fun", &"<FUNC>")
+ .field("name", &self.name)
+ .field("args", &self.args)
+ .field("return_field", &self.return_field)
+ .finish()
+ }
+}
+
+impl HigherOrderFunctionExpr {
+ /// Create a new Higher Order function
+ pub fn new(
+ name: impl Into<String>,
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ return_field: FieldRef,
+ config_options: Arc<ConfigOptions>,
+ ) -> Self {
+ Self {
+ fun,
+ name: name.into(),
+ args,
+ return_field,
+ config_options,
+ }
+ }
+
+ /// Create a new Higher Order function
+ pub fn try_new(
+ fun: Arc<dyn HigherOrderUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ config_options: Arc<ConfigOptions>,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_fields = args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(schema)?;
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `HigherOrderTypeSignature`
+ value_fields_with_higher_order_udf(&arg_fields, fun.as_ref())?;
+
+ let arguments = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| literal.value())
+ })
+ .collect::<Vec<_>>();
+
+ let ret_args = HigherOrderReturnFieldArgs {
+ arg_fields: &arg_fields,
+ scalar_arguments: &arguments,
+ };
+
+ let return_field = fun.return_field_from_args(ret_args)?;
+
+ Ok(Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ })
+ }
+
+ /// Get the higher order function implementation
+ pub fn fun(&self) -> &dyn HigherOrderUDF {
+ self.fun.as_ref()
+ }
+
+ /// The name for this expression
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ /// Input arguments
+ pub fn args(&self) -> &[Arc<dyn PhysicalExpr>] {
+ &self.args
+ }
+
+ /// Data type produced by this expression
+ pub fn return_type(&self) -> &DataType {
+ self.return_field.data_type()
+ }
+
+ pub fn with_nullable(mut self, nullable: bool) -> Self {
+ self.return_field = self
+ .return_field
+ .as_ref()
+ .clone()
+ .with_nullable(nullable)
+ .into();
+ self
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.return_field.is_nullable()
+ }
+
+ pub fn config_options(&self) -> &ConfigOptions {
+ &self.config_options
+ }
+}
+
+impl fmt::Display for HigherOrderFunctionExpr {
+ fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+ write!(f, "{}({})", self.name, expr_vec_fmt!(self.args))
+ }
+}
+
+impl PartialEq for HigherOrderFunctionExpr {
+ fn eq(&self, o: &Self) -> bool {
+ if std::ptr::eq(self, o) {
+ // The equality implementation is somewhat expensive, so let's short-circuit when possible.
+ return true;
+ }
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options,
+ } = self;
+ fun.eq(&o.fun)
+ && name.eq(&o.name)
+ && args.eq(&o.args)
+ && return_field.eq(&o.return_field)
+ && (Arc::ptr_eq(config_options, &o.config_options)
+ || sorted_config_entries(config_options)
+ == sorted_config_entries(&o.config_options))
+ }
+}
+impl Eq for HigherOrderFunctionExpr {}
+impl Hash for HigherOrderFunctionExpr {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ let Self {
+ fun,
+ name,
+ args,
+ return_field,
+ config_options: _, // expensive to hash, and often equal
+ } = self;
+ fun.hash(state);
+ name.hash(state);
+ args.hash(state);
+ return_field.hash(state);
+ }
+}
+
+fn sorted_config_entries(config_options: &ConfigOptions) -> Vec<ConfigEntry> {
+ let mut entries = config_options.entries();
+ entries.sort_by(|l, r| l.key.cmp(&r.key));
+ entries
+}
+
+impl PhysicalExpr for HigherOrderFunctionExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(self.return_field.data_type().clone())
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(self.return_field.is_nullable())
+ }
+
+ fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+ let arg_fields = self
+ .args
+ .iter()
+ .map(|e| {
+ let field = e.return_field(batch.schema_ref())?;
+
+ match e.as_any().downcast_ref::<LambdaExpr>() {
+ Some(_lambda) => Ok(ValueOrLambda::Lambda(field)),
+ None => Ok(ValueOrLambda::Value(field)),
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let value_fields = arg_fields
+ .iter()
+ .filter_map(|field| match field {
+ ValueOrLambda::Value(field) => Some(Arc::clone(field)),
+ ValueOrLambda::Lambda(_field) => None,
+ })
+ .collect::<Vec<_>>();
+
+ // lambda_parameters refers only to lambdas and not to values, so instead
+ // of zipping it with self.args, we iterate over self.args and only
+ // consume from lambda_parameters when a given argument is a lambda
+ // to reconstruct the arguments list with the correct order
+ // this supports any value and lambda positioning including
+ // multiple lambdas interleaved with values
+ let mut lambda_parameters =
+ self.fun().lambda_parameters(&value_fields)?.into_iter();
+ let num_lambdas = self.args.len() - value_fields.len();
+
+ // functions can support multiple lambdas where some trailing ones are optional,
+ // but to simplify the implementor, lambda_parameters returns the parameters of all of them,
+ // so we can't do equality check. one example is spark reduce:
+ // https://spark.apache.org/docs/latest/api/sql/index.html#reduce
+ if lambda_parameters.len() < num_lambdas {
+ return exec_err!(
+ "{} invocation defined {num_lambdas} but lambda_parameters returned only {}",
+ self.name(),
+ lambda_parameters.len()
+ );
+ }
+
+ let args = self
+ .args
+ .iter()
+ .map(|arg| match arg.as_any().downcast_ref::<LambdaExpr>() {
+ Some(lambda) => {
+ let lambda_params = lambda_parameters.next().ok_or_else(|| {
+ internal_datafusion_err!(
+ "params len should have been checked above"
+ )
+ })?;
+
+ if lambda.params().len() > lambda_params.len() {
+ return exec_err!(
+ "lambda defined {} params but UDF support only {}",
+ lambda.params().len(),
+ lambda_params.len()
+ );
+ }
+
+ let params = std::iter::zip(lambda.params(), lambda_params)
+ .map(|(name, param)| Arc::new(param.with_name(name)))
+ .collect();
+
+ Ok(ValueOrLambda::Lambda(LambdaArgument::new(
+ params,
+ Arc::clone(lambda.body()),
+ )))
+ }
+ None => {
+ let value = arg.evaluate(batch)?;
+
+ let value =
+ if self.fun.clear_null_values() && value.data_type().is_list() {
+ ColumnarValue::Array(remove_list_null_values(
+ &value.into_array(batch.num_rows())?,
+ )?)
+ } else {
+ value
+ };
+
+ Ok(ValueOrLambda::Value(value))
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let input_empty = args.is_empty();
+ let input_all_scalar = args
+ .iter()
+ .all(|arg| matches!(arg, ValueOrLambda::Value(ColumnarValue::Scalar(_))));
+
+ // evaluate the function
+ let output = self.fun.invoke_with_args(HigherOrderFunctionArgs {
+ args,
+ arg_fields,
+ number_rows: batch.num_rows(),
+ return_field: Arc::clone(&self.return_field),
+ config_options: Arc::clone(&self.config_options),
+ })?;
+
+ if let ColumnarValue::Array(array) = &output
+ && array.len() != batch.num_rows()
+ {
+ // If the arguments are a non-empty slice of scalar values, we can assume that
+ // returning a one-element array is equivalent to returning a scalar.
+ let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
+ return if preserve_scalar {
+ ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
+ } else {
+ internal_err!(
+ "UDF {} returned a different number of rows than expected. Expected: {}, Got: {}",
+ self.name,
+ batch.num_rows(),
+ array.len()
+ )
+ };
+ }
+ Ok(output)
+ }
+
+ fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
+ Ok(Arc::clone(&self.return_field))
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.args.iter().collect()
Review Comment:
https://github.com/gstvg/arrow-datafusion/blob/5c0b41df235a28ae508caad58b160b4c008b1d7d/datafusion/physical-expr/src/higher_order_function.rs#L245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org