You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/21 17:10:49 UTC

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

alamb commented on code in PR #10098:
URL: https://github.com/apache/datafusion/pull/10098#discussion_r1573855338


##########
datafusion/functions/src/math/coalesce.rs:
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I would personally expect to find coalesce in the `core` functions module rather than the `math` module, but we could do that as a follow on PR (or never).



##########
datafusion/expr/src/expr.rs:
##########
@@ -362,10 +362,6 @@ impl Between {
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
 /// Defines which implementation of a function for DataFusion to call.
 pub enum ScalarFunctionDefinition {
-    /// Resolved to a `BuiltinScalarFunction`
-    /// There is plan to migrate `BuiltinScalarFunction` to UDF-based implementation (issue#8045)
-    /// This variant is planned to be removed in long term
-    BuiltIn(BuiltinScalarFunction),
     /// Resolved to a user defined function
     UDF(Arc<crate::ScalarUDF>),
     /// A scalar function constructed with name. This variant can not be executed directly

Review Comment:
   Interestingly, I don't think Name is ever used anymore -- we could eventually simply remove `ScalarFunctionDefinition` to avoid another level of wrapping. 
   
   However, I think there has been enough API churn for a while. We can maybe clean that up as a follow on PR in a few months or somthing



##########
datafusion/proto/src/physical_plan/from_proto.rs:
##########
@@ -340,24 +339,10 @@ pub fn parse_physical_expr(
             convert_required!(e.arrow_type)?,
         )),
         ExprType::ScalarFunction(e) => {
-            let scalar_function =
-                protobuf::ScalarFunction::try_from(e.fun).map_err(|_| {
-                    proto_error(
-                        format!("Received an unknown scalar function: {}", e.fun,),
-                    )
-                })?;
-
-            let args = parse_physical_exprs(&e.args, registry, input_schema, codec)?;
-
-            // TODO Do not create new the ExecutionProps
-            let execution_props = ExecutionProps::new();
-
-            functions::create_builtin_physical_expr(
-                &(&scalar_function).into(),
-                &args,
-                input_schema,
-                &execution_props,
-            )?
+            return Err(proto_error(format!(

Review Comment:
   Maybe we could remove the ScalarFunction part of proto too 
   
   In a follow on PR we could probably remove 
   
   https://github.com/apache/datafusion/blob/70db5eab8996af4816958f798f6ee887dffb69ed/datafusion/proto/proto/datafusion.proto#L541-L686
   
   And the references to `ScalarFunctionNode`



##########
datafusion/expr/src/expr.rs:
##########
@@ -1276,7 +1260,7 @@ impl Expr {
     pub fn short_circuits(&self) -> bool {
         match self {
             Expr::ScalarFunction(ScalarFunction { func_def, .. }) => {
-                matches!(func_def, ScalarFunctionDefinition::BuiltIn(fun) if *fun == BuiltinScalarFunction::Coalesce)
+                matches!(func_def, ScalarFunctionDefinition::UDF(fun) if fun.name().eq("coalesce"))

Review Comment:
   I agree that @haohuaijin 's idea is a good one (and keeping with the spirit of "anything we can do with built in functions can be done with ScalarAPIs"). I filed https://github.com/apache/datafusion/issues/10162 to track
   



##########
datafusion/physical-expr/src/scalar_function.rs:
##########
@@ -144,24 +144,11 @@ impl PhysicalExpr for ScalarFunctionExpr {
     fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
         // evaluate the arguments, if there are no arguments we'll instead pass in a null array
         // indicating the batch size (as a convention)
-        let inputs = match (
-            self.args.is_empty(),
-            self.name.parse::<BuiltinScalarFunction>(),
-        ) {
-            // MakeArray support zero argument but has the different behavior from the array with one null.
-            (true, Ok(scalar_fun))
-                if scalar_fun
-                    .signature()
-                    .type_signature
-                    .supports_zero_argument() =>
-            {
-                vec![ColumnarValue::create_null_array(batch.num_rows())]
-            }
+        let inputs = match self.args.is_empty() {
             // If the function supports zero argument, we pass in a null array indicating the batch size.
             // This is for user-defined functions.
-            (true, Err(_))
-                if self.supports_zero_argument && self.name != "make_array" =>
-            {
+            // MakeArray support zero argument but has the different behavior from the array with one null.

Review Comment:
   This is a strange special case (that might be left over). @jayzhan211  or @Weijun-H  do you remember if this is still needed?



##########
datafusion/proto/src/logical_plan/from_proto.rs:
##########
@@ -412,16 +411,6 @@ impl From<&protobuf::StringifiedPlan> for StringifiedPlan {
     }
 }
 
-impl From<&protobuf::ScalarFunction> for BuiltinScalarFunction {

Review Comment:
   I think this is the correct change -- now all functions are resolved by name.
   
   While this is a breaking API change, most other changes to the protobuf definitions are breaking changes -- see https://github.com/apache/datafusion/blob/70db5eab8996af4816958f798f6ee887dffb69ed/datafusion/proto/src/lib.rs#L37-L42



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org