You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Toby1009 (via GitHub)" <gi...@apache.org> on 2026/09/23 21:18:02 UTC

[I] Preserve scalar UDF output types in property analysis with a central fallback [datafusion]

Toby1009 opened a new issue, #25667:
URL: https://github.com/apache/datafusion/issues/25667

   ### Is your feature request related to a problem or challenge?
   
   Scalar UDFs using the default `evaluate_bounds` return an unbounded `Null` interval. During physical property analysis, this loses the output type even though `ScalarFunctionExpr` already stores the resolved `return_field`. Parent expressions that inspect `range.data_type()` may therefore retain redundant sorts.
   
   #25575 addresses this for `date_bin` and `from_unixtime` with individual typed-bounds overrides. A central fallback could preserve type information for other built-in and user-defined scalar functions without requiring each function to implement `evaluate_bounds` solely for this purpose.
   
   ### Describe the solution you'd like
   
   Add a fallback in `ScalarFunctionExpr::get_properties` that uses the resolved return type to construct a typed unbounded interval when the UDF returns an unbounded `Null` interval.
   
   As discussed during review of #25575:
   
   - Replace only an unbounded `Null` result; preserve any bounds the UDF provides.
   - If `Interval::make_unbounded(return_type)` fails, retain the original `Null` interval rather than failing query planning. Errors from the UDF's own bounds evaluation should continue to propagate.
   - Leave `PhysicalExpr::evaluate_bounds` and its constraint-solver path unchanged.
   - Once equivalent behavior is covered, remove the type-only `evaluate_bounds` overrides added for `date_bin` and `from_unixtime`.
   
   Preserving a type does not itself establish ordering; existing expression-specific ordering rules must still apply.
   
   Validation should cover successful type recovery, preservation of existing bounds, unsupported interval types, and unchanged constraint-solver behavior. Retain positive plan tests for safe timezone-free compositions, and add negative plan tests showing that required sorts remain for named-timezone function outputs and time-of-day arithmetic with intervals through a function, including midnight wraparound.
   
   ### Describe alternatives you've considered
   
   - Continue adding typed-bounds overrides to individual UDFs.
   - Store `DataType` independently in `ExprProperties`. This would require a separate design discussion because it affects the public API, consumers, and FFI compatibility.
   
   ### Additional context
   
   Follow-up to #25575 and its [review discussion](https://github.com/apache/datafusion/pull/25575#issuecomment-5797397833).
   
   The [initial local prototype](https://github.com/apache/datafusion/pull/25575#issuecomment-5765234394) also observed redundant sort removal for `CAST(signum(c) AS REAL)` and `CAST(trunc(c) AS REAL)` over sorted inputs. These provide additional compositions to validate beyond the two datetime functions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve scalar UDF output types in property analysis with a central fallback [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #25667: Preserve scalar UDF output types in property analysis with a central fallback
URL: https://github.com/apache/datafusion/issues/25667


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Preserve scalar UDF output types in property analysis with a central fallback [datafusion]

Posted by "Toby1009 (via GitHub)" <gi...@apache.org>.
Toby1009 commented on issue #25667:
URL: https://github.com/apache/datafusion/issues/25667#issuecomment-5803099103

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org