You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2025/01/12 09:25:17 UTC
[PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
jayzhan211 opened a new pull request, #14094:
URL: https://github.com/apache/datafusion/pull/14094
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Part of #13717
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1921009149
##########
datafusion/functions/src/core/coalesce.rs:
##########
@@ -174,39 +177,3 @@ fn get_coalesce_doc() -> &'static Documentation {
.build()
})
}
-
-#[cfg(test)]
Review Comment:
It is covered in slt, remove duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2599716451
> I plan to change to ScalarValue after #14167, so we can parse the string easily.
>
> If anyone interested in this change can also take it.
I have just merged in https://github.com/apache/datafusion/pull/14167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2585751923
> I am also not sure about only supporting string args, that is likely a regression in behavior for some users (For example, maybe they look for constant integers as well)
Yes, it might be, since I assume we don't really need `Expr` but `String` instead. However, in theory they can achieve what they need with `String`, but of course this is breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2603294891
Love it. And love that this isn't an API change. Thank you so much @jayzhan211 -- this really looks great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #14094:
URL: https://github.com/apache/datafusion/pull/14094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2587976364
> Do we really need Planning? My thought is that whenever we have Expr and Schema we can compute corresponding DataType and Nullability. Therefore, even for Planning stage, we can still get the information DataType + Nullability
I think the usecase is for expressions like `arrow_cast` whose output *type* depends on the actual value of the arguments (not just the type of the argument)
So like `arrow-cast(String)` can return `Int64` or `Float64` depending on the value of the argument -- the value is passed as an `Expr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1958070497
##########
datafusion/expr/src/udf.rs:
##########
@@ -481,15 +563,15 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
/// This function must consistently return the same type for the same
/// logical input even if the input is simplified (e.g. it must return the same
/// value for `('foo' | 'bar')` as it does for ('foobar').
- fn return_type_from_exprs(
- &self,
- _args: &[Expr],
- _schema: &dyn ExprSchema,
- arg_types: &[DataType],
- ) -> Result<DataType> {
- self.return_type(arg_types)
+ fn return_type_from_args(&self, args: ReturnTypeArgs) -> Result<ReturnInfo> {
+ let return_type = self.return_type(args.arg_types)?;
Review Comment:
This change was a bit problematic for users with custom UDFs in that it may cause breaks at runtime.
The `#[deprecated()]` apparently doesn't affect for people _implementing_ this trait, only those who call it. As such, if someone (like us) had implemented `return_type_from_exprs` and had `return_type` just panic, that'd compile fine, but rather than calling the `return_type_from_exprs` DF would call this new `return_type_from_args` with its default impl delegating to `return_type` which would then panic.
Luckily, we had tests that caught it, but in general I think in this case it would have been better to fully remove the `return_type_from_exprs` so that users who override it would get compile failures, rather than runtime ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1921677988
##########
datafusion/common/src/utils/mod.rs:
##########
@@ -1201,4 +1202,13 @@ mod tests {
assert_eq!(expected, transposed);
Ok(())
}
+
+ #[test]
+ fn test132() {
+ let a = Arc::new(Int32Array::from(vec![3; 200])) as ArrayRef;
+ println!(
+ "display {}",
+ pretty_format_columns("ColumnarValue(ArrayRef)", &[a]).unwrap()
Review Comment:
Oh, I forgot to remove this 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2597031931
This is on my review queue, I just haven't quite had time to get to it yet. I will do so tomorrow if not later tonight
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2588588440
Or we could merge in this approach and add back in Expr support prior to the 45 release 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2599625699
I plan to change to ScalarValue after #14167, so we can parse the string easily.
If anyone interested in this change can also take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1912406060
##########
datafusion/functions/src/core/named_struct.rs:
##########
@@ -44,11 +44,18 @@ fn named_struct_expr(args: &[ColumnarValue]) -> Result<ColumnarValue> {
.chunks_exact(2)
.enumerate()
.map(|(i, chunk)| {
-
let name_column = &chunk[0];
let name = match name_column {
- ColumnarValue::Scalar(ScalarValue::Utf8(Some(name_scalar))) => name_scalar,
- _ => return exec_err!("named_struct even arguments must be string literals, got {name_column:?} instead at position {}", i * 2)
Review Comment:
`name_column` output less readable array in this change, remove it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2585752891
```rust
#[derive(Debug)]
pub enum ReturnTypeArgs<'a> {
/// information known at logical planning time
/// Note you can get get type and nullability for each arg
// using the specified ExprSchema
Planning {
pub args: &'a[Expr],
pub schema: &'a dyn ExprSchema
},
/// Information known during Execution
Execution {
/// The data types of the arguments to the function
pub arg_types: &'a [DataType],
pub arg_nullability: [bool],
}
}
```
One good thing in this PR is that we don't need `Expr` anymore, we compute data type and nullable in datafusion core and they are not "public" for customization.
Do we really need `Planning`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1958192561
##########
datafusion/expr/src/udf.rs:
##########
@@ -481,15 +563,15 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
/// This function must consistently return the same type for the same
/// logical input even if the input is simplified (e.g. it must return the same
/// value for `('foo' | 'bar')` as it does for ('foobar').
- fn return_type_from_exprs(
- &self,
- _args: &[Expr],
- _schema: &dyn ExprSchema,
- arg_types: &[DataType],
- ) -> Result<DataType> {
- self.return_type(arg_types)
+ fn return_type_from_args(&self, args: ReturnTypeArgs) -> Result<ReturnInfo> {
+ let return_type = self.return_type(args.arg_types)?;
Review Comment:
Filed https://github.com/apache/datafusion/issues/14729.
`invoke_batch` doesn't suffer from this runtime issue since `invoke_with_args` by default delegates to `invoke_batch`, so it is enough for users to still just override `invoke_batch`.
It does however have the same issue in that the users who do override `invoke_batch` aren't alerted to it being deprecated, meaning its future removal will still be a sudden compile-time break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1912405518
##########
datafusion/expr/src/udf.rs:
##########
@@ -209,10 +214,15 @@ impl ScalarUDF {
self.inner.invoke(args)
}
+ #[allow(deprecated)]
pub fn is_nullable(&self, args: &[Expr], schema: &dyn ExprSchema) -> bool {
self.inner.is_nullable(args, schema)
}
+ pub fn is_nullable_from_args_nullable(&self, args_nullables: &[bool]) -> bool {
Review Comment:
Remove `Expr` dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1912405731
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +352,14 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
+ /// The data types of the arguments to the function
+ pub arg_types: &'a [DataType],
+ /// The Utf8 arguments to the function, if the expression is not Utf8, it will be empty string
+ pub arguments: &'a [String],
Review Comment:
better name 🤔 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2585670790
As stated in https://github.com/apache/datafusion/pull/13717#discussion_r1888341956 , this new method doesn't necessarily simplify anything.
Can you please fill "Rationale for this change"? What problem are we solving?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2585750558
> Multiple APIs (Nullability and return type)
Great, I also want this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1920358041
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +348,56 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
Review Comment:
Here are some suggestions to improve the comments:
```suggestion
/// Arguments passed to [`ScalarUDFImpl::return_type_from_args`]
#[derive(Debug)]
pub struct ReturnTypeArgs<'a> {
```
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +348,56 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
+ /// The data types of the arguments to the function
+ pub arg_types: &'a [DataType],
+ /// The Utf8 arguments to the function, if the expression is not Utf8, it will be empty string
Review Comment:
If possible I suggest using the following type:
1. It can distinguish between string/non-string arguments
3. It avoids clones
```rust
pub arguments: &'a [Option<&'a ScalarValue>]>`,
```
##########
datafusion/expr/src/udf.rs:
##########
@@ -490,6 +547,40 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
self.return_type(arg_types)
}
+ /// What [`DataType`] will be returned by this function, given the
Review Comment:
I also recommend removing the comments from `return_type_from_exprs` now that it is deprecated and you have moved the content here
##########
datafusion/expr/src/udf.rs:
##########
@@ -481,6 +537,7 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
/// This function must consistently return the same type for the same
/// logical input even if the input is simplified (e.g. it must return the same
/// value for `('foo' | 'bar')` as it does for ('foobar').
+ #[deprecated(since = "45.0.0", note = "Use `return_type_from_args` instead")]
Review Comment:
Can you also please update the comments on `return_type` to refer to `return_type_from_args` (it currently talks about `return_type_from_exprs`:
```rust
/// If you provide an implementation for [`Self::return_type_from_exprs`],
/// DataFusion will not call `return_type` (this function). In this case it
/// is recommended to return [`DataFusionError::Internal`].
```
##########
datafusion/functions/src/core/named_struct.rs:
##########
@@ -44,11 +44,18 @@ fn named_struct_expr(args: &[ColumnarValue]) -> Result<ColumnarValue> {
.chunks_exact(2)
.enumerate()
.map(|(i, chunk)| {
-
let name_column = &chunk[0];
let name = match name_column {
- ColumnarValue::Scalar(ScalarValue::Utf8(Some(name_scalar))) => name_scalar,
- _ => return exec_err!("named_struct even arguments must be string literals, got {name_column:?} instead at position {}", i * 2)
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(name_scalar))) => {
+ name_scalar
+ }
+ // TODO: Implement Display for ColumnarValue
Review Comment:
That is a good todo -- maybe we can make a follow on ticket for it
##########
datafusion/functions/src/core/coalesce.rs:
##########
@@ -174,39 +177,3 @@ fn get_coalesce_doc() -> &'static Documentation {
.build()
})
}
-
-#[cfg(test)]
Review Comment:
why remove these tests?
##########
datafusion/physical-expr/src/scalar_function.rs:
##########
@@ -83,6 +86,53 @@ impl ScalarFunctionExpr {
}
}
+ /// Create a new Scalar function
+ pub fn try_new(
+ fun: Arc<ScalarUDF>,
+ args: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &Schema,
+ ) -> Result<Self> {
+ let name = fun.name().to_string();
+ let arg_types = args
+ .iter()
+ .map(|e| e.data_type(schema))
+ .collect::<Result<Vec<_>>>()?;
+
+ // verify that input data types is consistent with function's `TypeSignature`
+ data_types_with_scalar_udf(&arg_types, &fun)?;
+
+ let nullables = args
+ .iter()
+ .map(|e| e.nullable(schema))
+ .collect::<Result<Vec<_>>>()?;
+
+ let arguments: Vec<String> = args
+ .iter()
+ .map(|e| {
+ e.as_any()
+ .downcast_ref::<Literal>()
+ .map(|literal| match literal.value() {
+ ScalarValue::Utf8(Some(s)) => s.clone(),
Review Comment:
It would be really nice to avoid this clone here -- if we changed the signature to `&[Option<&str>]` I think that would be easier to use and more efficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2588854022
> Are you ok with passing `Expr`s in theory but just don't have time to make it work
No, actually the logical concept is what I want to avoid and I think this is unnecessary, having computed properties are what we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1912405839
##########
datafusion/functions/src/core/arrow_cast.rs:
##########
@@ -86,22 +87,36 @@ impl ScalarUDFImpl for ArrowCastFunc {
}
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
Review Comment:
If this looks good, we can deprecate this too
##########
datafusion/functions/src/core/arrow_cast.rs:
##########
@@ -86,22 +87,36 @@ impl ScalarUDFImpl for ArrowCastFunc {
}
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
Review Comment:
If this change looks good, we can deprecate this too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add two new methods in ScalarFunction `return_type_from_args` and `is_nullable_from_args_nullable` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1912432276
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +352,14 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
+ /// The data types of the arguments to the function
+ pub arg_types: &'a [DataType],
+ /// The Utf8 arguments to the function, if the expression is not Utf8, it will be empty string
+ pub arguments: &'a [String],
Review Comment:
I am also not sure about only supporting string args, that is likely a regression in behavior for some users (For example, maybe they look for constant integers as well)
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +352,14 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
+ /// The data types of the arguments to the function
+ pub arg_types: &'a [DataType],
+ /// The Utf8 arguments to the function, if the expression is not Utf8, it will be empty string
+ pub arguments: &'a [String],
Review Comment:
Would it be possible to unify the argument handling so that both return type and nullability are returned the same?
I wonder if it would somehow be possible to add the input nullable information here too 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1958161993
##########
datafusion/expr/src/udf.rs:
##########
@@ -481,15 +563,15 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
/// This function must consistently return the same type for the same
/// logical input even if the input is simplified (e.g. it must return the same
/// value for `('foo' | 'bar')` as it does for ('foobar').
- fn return_type_from_exprs(
- &self,
- _args: &[Expr],
- _schema: &dyn ExprSchema,
- arg_types: &[DataType],
- ) -> Result<DataType> {
- self.return_type(arg_types)
+ fn return_type_from_args(&self, args: ReturnTypeArgs) -> Result<ReturnInfo> {
+ let return_type = self.return_type(args.arg_types)?;
Review Comment:
> Luckily, we had tests that caught it, but in general I think in this case it would have been better to fully remove the return_type_from_exprs so that users who override it would get compile failures, rather than runtime ones.
THis makes sense to me - can you file a ticket? I think this one is related but I am not sure it is entirely the same
- https://github.com/apache/datafusion/issues/14652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2588877877
For example, Comet don't have logical expression `Expr` but if they want to create physical scalar function, they need to somehow create `Expr` in order to build a physical scalar function. However, the metadata of their logical plan's properties like arguments data type and nullability, those **arguments metadata** are what really matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2588488394
> So like `arrow-cast(String`
For arrow_cast, the output type is determined by the 2nd argument which is string.
arrow_cast(a, 'Int64') returns I64, arrow_cast(a, 'List(Int64)') returns List(Int64)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2588587414
> All the functions in datafusion have constant string type, the only concern is that other users do the crazy things with `Expr`, but I don't think it is practical since String is enough in my opionion
I don't have a strong opinion here. Strictly speaking only supporting strings rather than Exprs is a regression in functionality. Are you ok with passing `Expr`s in theory but just don't have time to make it work? Maybe I can find some time to help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#discussion_r1921010901
##########
datafusion/expr/src/udf.rs:
##########
@@ -342,6 +348,56 @@ pub struct ScalarFunctionArgs<'a> {
pub return_type: &'a DataType,
}
+#[derive(Debug)]
+pub struct ReturnTypeArgs<'a> {
+ /// The data types of the arguments to the function
+ pub arg_types: &'a [DataType],
+ /// The Utf8 arguments to the function, if the expression is not Utf8, it will be empty string
Review Comment:
I'm fine with ScalarValue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14094:
URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2598845894
FYI @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org