You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "joroKr21 (via GitHub)" <gi...@apache.org> on 2024/10/14 15:25:50 UTC

[PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

joroKr21 opened a new pull request, #12922:
URL: https://github.com/apache/datafusion/pull/12922

   If all UDF arguments are scalars, so should be the result. In most cases, such function calls will be contant-folded, however if for whatever reason the are not optimized, we want to avoid an error due to array length mismatch.
   
   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   No particular issue, as I couldn't find a case that is not constant-folded with pure DataFusion, however we have some custom optimizer rules that might result in expressions that are not constant-folded.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   The rationale is that the UDF implementations shouldn't rely on optimizer rules to work correctly.
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   * Added a new helper function `ColumnarValue::from_args_and_result` to perform the check and conversion.
   * Updated the math UDF macros to return a scalar value when all their arguments are scalar.
   
   ## Are these changes tested?
   
   Relying on existing tests to ensure nothing breaks.
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   I made `ColumnarValue::from_args_and_result` public but if that's not desirable, I can move it to the math module.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802414188


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > This can be a general solution. The disadvantage is that if the arguments are not real scalar, it will cause unnecessary overhead, as the final result of queries will still need to be converted back into arrays.
   
   Yeah, but it sound very unlikely that we will have a batch of one row. In any case, we can do the same check that all arguments are scalar πŸ‘ 



##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > This can be a general solution. The disadvantage is that if the arguments are not real scalar, it will cause unnecessary overhead, as the final result of queries will still need to be converted back into arrays.
   
   Yeah, but it sounds very unlikely that we will have a batch of one row. In any case, we can do the same check that all arguments are scalar πŸ‘ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801708934


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Yeah anyway, I was just trying to show that this is a deficiency of the UDF model. Ideally, `invoke` should also get the number of expected rows and then there would also be no need for a special `invoke_no_args`.
   
   Regarding adding a flag with default `false`, that doesn't sound very useful. Maybe we can just assume that if the result of any immutable function is an array of size one, we might as well convert it to a scalar value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802220611


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > Ideally, `invoke` should also get the number of expected rows and then there would also be no need for a special `invoke_no_args`.
   
   Make sense to me. This makes it easy to support `rand_int`. It seems that currently all functions should return a scalar value when all their arguments are scalar, for example, math functions that do not use `make_math_unary_udf`. 
   
   > Maybe we can just assume that if the result of any immutable function is an array of size one, we might as well convert it to a scalar value.
   
   This can be a general solution. The disadvantage is that if the arguments are not real scalar, it will cause unnecessary overhead, as the final result of queries will still need to be converted back into arrays.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801303443


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   My thought is to add a new method, `fn maintains_scalar() -> bool`, to the `ScalarUDFImpl` trait, which by default returns false, with some math functions returning true. Or add such a flag in `Signature`. Check such the flag and apply similar processing as in this PR to all functions in one place.



##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   My thought is to add a new method, `fn maintains_scalar() -> bool`, to the `ScalarUDFImpl` trait, which by default returns false, with some math functions returning true. Or add such a flag in `Signature`. Check the flag and apply similar processing as in this PR to all functions in one place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801248305


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   The result after invoking is a `ColumnarValue`, and we can determine the number of rows .If a function might produce multiple rows for scalar args, it might not be appropriate to declare it as maintain_scalar πŸ€”.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801368848


##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -218,6 +217,17 @@ impl ColumnarValue {
             }
         }
     }
+
+    /// Converts an [`ArrayRef`] to a [`ColumnarValue`] based on the supplied arguments.
+    /// This is useful for scalar UDF implementations to fulfil their contract:
+    /// if all arguments are scalar values, the result should also be a scalar value.
+    pub fn from_args_and_result(args: &[Self], result: ArrayRef) -> Result<Self> {
+        if result.len() == 1 && args.iter().all(|arg| matches!(arg, Self::Scalar(_))) {
+            Ok(Self::Scalar(ScalarValue::try_from_array(&result, 0)?))
+        } else {
+            Ok(Self::Array(result))

Review Comment:
   It already exists in constant folding, but we shouldn't rely on constant folding for the correct function implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801560858


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I understand your example: you want each invocation to use the same upper bound for multiple rows, and the result of each row is random. Itβ€˜s a special scalar function. I am not sure if there is a better way, maybe adding an extra placeholder argument to indicate the number of rows, for example: `select rand_int(10, a) from t`;



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801372082


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   So what I'm trying to say is imagine I want to implement a volatile function `rand_int(max: Int64) -> Int64`. How can I implement it correctly so that given a `ScalarValue` for `max` it will actually produce an array of random integers with the expected length?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802459241


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > Yeah, but it sounds very unlikely that we will have a batch of one row.
   
   it's unlikely, but you must not assume that.
   
   a trivial example is a source table with one row.
   
   > we can do the same check that all arguments are scalar πŸ‘
   
   πŸ‘ 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1799857700


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Note that @simonvandel and @jonahgao  have been removing use of `make_math_unary_udf` in general
   
   (or example https://github.com/apache/datafusion/pull/12889)
   
   We should probably make sure we aren't missing a similar problem in their reviews



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801221800


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   If a function needs to maintain scalar, we can call `ColumnarValue::from_args_and_result` after invoking it. 
   ```rust
           // in file: datafusion/physical-expr/src/scalar_function.rs
           // evaluate the function
           let output = match self.args.is_empty() {
               true => self.fun.invoke_no_args(batch.num_rows()),
               false => self.fun.invoke(&inputs),
           }?;
   
           if let ColumnarValue::Array(array) = &output {
               if self.fun.need_maintain_scalar() && !inputs.is_empty() {
                   output = ColumnarValue::from_args_and_result(inputs, array)
               }
           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801257537


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I think some other functions might also need to be handled similarly to this PR, so perhaps consider introducing a more general approach. Let's merge this PR first, and we can consider this issue further later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801446533


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Additionally, if I understand correctly, `rand_int` seems more like a table function, and supporting it in expression positions is not simple, similar to `unnset`, which I think goes beyond the scope of this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801226545


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I mean that if the function is volatile but it has only scalar arguments, how does it know how many rows to produce?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802220611


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > Ideally, `invoke` should also get the number of expected rows and then there would also be no need for a special `invoke_no_args`.
   
   Make sense to me. This makes it easy to support `rand_int`. It seems that currently all functions should return a scalar value when all their arguments are scalar, for example, math functions that do not use `make_math_*_udf`. 
   
   > Maybe we can just assume that if the result of any immutable function is an array of size one, we might as well convert it to a scalar value.
   
   This can be a general solution. The disadvantage is that if the arguments are not real scalar, it will cause unnecessary overhead, as the final result of queries will still need to be converted back into arrays.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao merged PR #12922:
URL: https://github.com/apache/datafusion/pull/12922


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801303443


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   My thought is to add a new method, `fn maintains_scalar() -> bool`, to the `ScalarUDFImpl` trait, which by default returns false, with some math functions returning true. Or add such a flag in `Signature`. Check such a flag and apply similar processing as in this PR to all functions in one place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801421236


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I guess the function `rand_int` works similarly to `unnest(range(1, arg))`. 
   I think we don't need to fulfill the ColumnarValue contract for this function if we define the contract as always returning a scalar for scalar args.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801521103


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   According to the definition of a scalar function, it produces a single row output for each row of input.
   https://github.com/apache/datafusion/blob/4a0b7688074880767e1e032400c1e97b7ffcb76e/datafusion/expr/src/udf.rs#L37
   
   I think this example should use `invoke`, and the number of rows produced is the same as the number of rows in args. The max arg of the i-th output row is taken from the i-th row of args[0].
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1800786590


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Is it necessary to add an attribute like `maintains_scalar` to `ScalarUDFImpl` and then automatically enforce this process during [evaluation](https://github.com/apache/datafusion/blob/d9450da6991a2977c617dfa789f208c53ba11421/datafusion/physical-expr/src/scalar_function.rs#L145)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1800929476


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I guess it's not necessarily true for volatile functions. But I wonder then how will the function know the array length? E.g. for a `rand(max)` function if I call it like `rand(42)` which is pretty reasonable, how does it work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801330973


##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -218,6 +217,17 @@ impl ColumnarValue {
             }
         }
     }
+
+    /// Converts an [`ArrayRef`] to a [`ColumnarValue`] based on the supplied arguments.
+    /// This is useful for scalar UDF implementations to fulfil their contract:
+    /// if all arguments are scalar values, the result should also be a scalar value.
+    pub fn from_args_and_result(args: &[Self], result: ArrayRef) -> Result<Self> {
+        if result.len() == 1 && args.iter().all(|arg| matches!(arg, Self::Scalar(_))) {
+            Ok(Self::Scalar(ScalarValue::try_from_array(&result, 0)?))
+        } else {
+            Ok(Self::Array(result))

Review Comment:
   If we allow such code to exist, wouldn't it be better to just have it during constant-folding, rather than inside function implementations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801484804


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Well simply the issue is that I don't know what is the batch size expected to return from this function. Usually it's inferred by any of the argument's length. But if all arguments are scalar values, then I can return a scalar value and all is well. This works for immutable functions but not for volatile ones. See e.g.
   
   ```rust
   impl ScalarUDFImpl for RandomFunc {
       fn as_any(&self) -> &dyn Any {
           self
       }
   
       fn name(&self) -> &str {
           "random"
       }
   
       fn signature(&self) -> &Signature {
           &self.signature
       }
   
       fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
           Ok(Float64)
       }
   
       fn invoke(&self, _args: &[ColumnarValue]) -> Result<ColumnarValue> {
           not_impl_err!("{} function does not accept arguments", self.name())
       }
   
       fn invoke_no_args(&self, num_rows: usize) -> Result<ColumnarValue> {
           let mut rng = thread_rng();
           let mut values = vec![0.0; num_rows];
           // Equivalent to set each element with rng.gen_range(0.0..1.0), but more efficient
           rng.fill(&mut values[..]);
           let array = Float64Array::from(values);
   
           Ok(ColumnarValue::Array(Arc::new(array)))
       }
   }
   ```
   
   What if I wanted to implement `RandomFunc` with an upper bound? I can't use `invoke_no_args` because I have one argument and I can't use `invoke` because it doesn't tell me how many rows I should produce.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801484804


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Well simply the issue is that I don't know what is the batch size expected to return from this function. Usually it's inferred by any of the argument's length. But if all arguments are scalar values, then I can return a scalar value and all is well. This works for immutable functions but not for volatile ones. See e.g.
   
   ```rust
   impl ScalarUDFImpl for RandomFunc {
       fn as_any(&self) -> &dyn Any {
           self
       }
   
       fn name(&self) -> &str {
           "random"
       }
   
       fn signature(&self) -> &Signature {
           &self.signature
       }
   
       fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
           Ok(Float64)
       }
   
       fn invoke(&self, _args: &[ColumnarValue]) -> Result<ColumnarValue> {
           not_impl_err!("{} function does not accept arguments", self.name())
       }
   
       fn invoke_no_args(&self, num_rows: usize) -> Result<ColumnarValue> {
           let mut rng = thread_rng();
           let values = std::iter::repeat_with(|| rng.gen_range(0.0..1.0)).take(num_rows);
           let array = Float64Array::from_iter_values(values);
           Ok(ColumnarValue::Array(Arc::new(array)))
       }
   }
   ```
   
   What if I wanted to implement `RandomFunc` with an upper bound? I can't use `invoke_no_args` because I have one argument and I can't use `invoke` because it doesn't tell me how many rows I should produce.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801521103


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   According to the definition of a scalar function, it produces a single row output for each row of input.
   https://github.com/apache/datafusion/blob/4a0b7688074880767e1e032400c1e97b7ffcb76e/datafusion/expr/src/udf.rs#L37
   
   I think this example should use `invoke`, and the number of rows produced is the same as the number of rows in args. The upper bound arg of the i-th output row is taken from the i-th row of args[0].
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#issuecomment-2415568512

   Thanks @joroKr21  Let's merge it first, and then consider similar problem for other scalar functions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802530293


##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -218,6 +217,17 @@ impl ColumnarValue {
             }
         }
     }
+
+    /// Converts an [`ArrayRef`] to a [`ColumnarValue`] based on the supplied arguments.
+    /// This is useful for scalar UDF implementations to fulfil their contract:
+    /// if all arguments are scalar values, the result should also be a scalar value.
+    pub fn from_args_and_result(args: &[Self], result: ArrayRef) -> Result<Self> {
+        if result.len() == 1 && args.iter().all(|arg| matches!(arg, Self::Scalar(_))) {
+            Ok(Self::Scalar(ScalarValue::try_from_array(&result, 0)?))
+        } else {
+            Ok(Self::Array(result))

Review Comment:
   It's not the only place where it matters. If for some reason the expression is not constant folded, you will get an error that the array lengths of different columns don't match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801484804


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   Well simply the issue is that I don't know how what is the batch size expected to return from this function. Usually it's inferred by any of the argument's length. But if all arguments are scalar values, then I can return a scalar value and all is well. This works for immutable functions but not for volatile ones. See e.g.
   
   ```rust
   impl ScalarUDFImpl for RandomFunc {
       fn as_any(&self) -> &dyn Any {
           self
       }
   
       fn name(&self) -> &str {
           "random"
       }
   
       fn signature(&self) -> &Signature {
           &self.signature
       }
   
       fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
           Ok(Float64)
       }
   
       fn invoke(&self, _args: &[ColumnarValue]) -> Result<ColumnarValue> {
           not_impl_err!("{} function does not accept arguments", self.name())
       }
   
       fn invoke_no_args(&self, num_rows: usize) -> Result<ColumnarValue> {
           let mut rng = thread_rng();
           let values = std::iter::repeat_with(|| rng.gen_range(0.0..1.0)).take(num_rows);
           let array = Float64Array::from_iter_values(values);
           Ok(ColumnarValue::Array(Arc::new(array)))
       }
   }
   ```
   
   What if I wanted to implement `RandomFunc` with an upper bound? I can't use `invoke_no_args` because I have one argument and I can't use `invoke` because it doesn't tell me how many rows I should produce.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802461413


##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -218,6 +217,17 @@ impl ColumnarValue {
             }
         }
     }
+
+    /// Converts an [`ArrayRef`] to a [`ColumnarValue`] based on the supplied arguments.
+    /// This is useful for scalar UDF implementations to fulfil their contract:
+    /// if all arguments are scalar values, the result should also be a scalar value.
+    pub fn from_args_and_result(args: &[Self], result: ArrayRef) -> Result<Self> {
+        if result.len() == 1 && args.iter().all(|arg| matches!(arg, Self::Scalar(_))) {
+            Ok(Self::Scalar(ScalarValue::try_from_array(&result, 0)?))
+        } else {
+            Ok(Self::Array(result))

Review Comment:
   That depends how we define "correct".
   IF constant folding is the only party where maintaining ColumnarValue::Scalar matters, THEN we can redefine "correct", loosening the contract. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801257537


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   I think some other functions might also be handled like in this PR, so perhaps consider introducing a more general approach. Let's merge this PR first, and maybe we can consider this issue further later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "joroKr21 (via GitHub)" <gi...@apache.org>.
joroKr21 commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1801270692


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   > If a function might produce multiple rows for scalar args, it might not be appropriate to declare it as maintain_scalar
   
   Yes, but how will you actually implement it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Ensure that math functions fulfil the ColumnarValue contract [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #12922:
URL: https://github.com/apache/datafusion/pull/12922#discussion_r1802231687


##########
datafusion/functions/src/macros.rs:
##########
@@ -226,9 +226,8 @@ macro_rules! make_math_unary_udf {
                     $EVALUATE_BOUNDS(inputs)
                 }
 
-                fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
-                    let args = ColumnarValue::values_to_arrays(args)?;
-
+                fn invoke(&self, col_args: &[ColumnarValue]) -> Result<ColumnarValue> {

Review Comment:
   @joroKr21 @alamb Filed a new issue for other functions #12959



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org