Re: [I] Feedback request for providing configurable UDF functions [datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 commented on issue #10744:
URL: https://github.com/apache/datafusion/issues/10744#issuecomment-2143199323

   I think it is possible to extend the `safe_mode` to the datafusion core like what you mentioned, it should be similar to the `distinct` mode to the aggregate function. We can have different behavior based on whether "safe_mode" is set. 
   
   For the expression API, we can either 
   
   1. Introduce `to_timestamp_safe()`
   2. Extend `to_timestamp(args, safe: bool)`
   3. Introduce builder mode to  avoid breaking change `to_timestamp(args).safe().build()`
   
   I prefer the third one.
   
   Also, there are many functions with different time units, I'm not sure why they are split into different functions, they are possible to collapse into a single to_timestamp function and cast to different time units based on the given argument.
   
   We can have `to_timestamp(args).time_unit(Mili).safe().build()` if we need the timestamp millisecond.
   
   > "will safemode to_timestamp be part of the datafusion core"
   
   It is an interesting question, we can think of implementing functions based on other DB in the first place.
   
   For example, we usually follow postgres, duckdb, and others.
   
   We can have `functions-postgres`, `functions-duckdb` and `functions-spark` that aim to mirror the behavior of other db, and we don't even need `functions` crate (we can keep them for backward compatibility). They are considered as the `extension crate implemented by the third party`, datafusion does not need to implement any datafusion-specific function (and we prefer not to), we just make sure datafusion core is possible compatible with different functions crate. And we can register most-used functions to the datafusion for the end-to-end SQL workflow!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org