Re: [I] Support smaller decimal types through SQL interface [datafusion]

["AdamGS (via GitHub)" <gi...@apache.org>]

AdamGS commented on issue #17747:
URL: https://github.com/apache/datafusion/issues/17747#issuecomment-3401227536

   There has been a bunch of work to improve decimal support (by me and others), but the SQL part remains open. My current understanding and thinking here is:
   1. Supporting decimal32/64 through the SQL interface is a breaking change. Even though the current implementation makes it easy to use them together, there are cases where that doesn't currently work (The most obvious one I've run into is the Spark `mod` function).
   2. Making it work will probably require changes to type coercion and how functions operating on decimals determine their output type.
   3. I know there are people thinking/working on changing DataFusion's type system to a more logical one that abstracts away the arrow type system. This seems like a good fit to other SQL systems, as they usually treat decimal (or numeric as its often called) as effectively a logical type, determining its physical size by the actual data it contains.
   
   If 3 actually happens, it'll make this much easier. Otherwise - I think we can put it behind some config flag, which is what was done for `Utf8View` and it was introduced and replaced `Utf8` as the default string type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org