[I] Cache the arguments type to avoid recomputing for ScalarUDF::return_type_from_exprs [arrow-datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 opened a new issue, #9518:
URL: https://github.com/apache/arrow-datafusion/issues/9518

   ### Is your feature request related to a problem or challenge?
   
   In #9504, we have an issue that `select array_ndims([[[[[[[[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]]]]]]])` is taking so long time compare with the main branch.
   
   I found out that we always calculate the arguments type every time, which is really costly for the expression like above.
   
   https://github.com/apache/arrow-datafusion/blob/5537572820977b38719e2253f601a159deef5bc6/datafusion/expr/src/expr_schema.rs#L141-L156
   
   UDF now always go through `return_type_from_exprs`
   
   https://github.com/apache/arrow-datafusion/blob/5537572820977b38719e2253f601a159deef5bc6/datafusion/expr/src/udf.rs#L306-L316
   
   Always recompute types here.
   
   I test it locally that every avoid the re-computation, at least this query gets super fast as the main branch does.
   
   ### Describe the solution you'd like
   
   Ideally, we should cache the result for the same arguments and schema pair, but that need to check the matching first. 
   I think we can just add another argument `args_types: &[DataType]` for `return_type_from_exprs`. If the `args_types` is empty, we calculate the types, otherwise we skip it.
   
   ### Describe alternatives you've considered
   
   Other better design
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Cache the arguments type to avoid recomputing for ScalarUDF::return_type_from_exprs [arrow-datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 closed issue #9518: Cache the arguments type to avoid recomputing for ScalarUDF::return_type_from_exprs
URL: https://github.com/apache/arrow-datafusion/issues/9518


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org