You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "theirix (via GitHub)" <gi...@apache.org> on 2025/11/23 13:26:16 UTC
[I] [EPIC] Support Decimal for User Defined Functions [datafusion]
theirix opened a new issue, #18889:
URL: https://github.com/apache/datafusion/issues/18889
### Is your feature request related to a problem or challenge?
Math UDFs in the [datafusion-functions crate](https://github.com/apache/datafusion/tree/main/datafusion/functions) support integer and floating types, but not Decimals. This epic is dedicated to adding and improving decimal support for UDFs. It is a follow-up epic to adding Decimal support to the DataFusion core #3523.
So far, it is implemented for `log`, `power` binary UDFs; `round`, `ceil` unary UDF. Turns out, more code should be moved into helper functions to make UDFs leaner and abstracted from details (e.g., scalar vs. array cases, casting, etc).
### Describe the solution you'd like
There are the following primary directions:
1) Adding support for well-known `Decimal128` and `Decimal256` to existing functions
2) Adding support for new `Decimal32` and `Decimal64`, which are not yet fully supported
3) Refining coercion rules to work with mixtures of floats/decimals
4) Ensuring it would work properly with the new `parse_float_as_decimal` flag, forcing floats to be decimals after SQL parsing
5) Improving tests to validate correct behaviour for floats/decimals and corner cases
6) Moving some core support to the Arrow libraries
I welcome thoughts and discussions about these directions.
### Describe alternatives you've considered
The approach of coercing decimals to floats could work, but it loses precision and data and doesn't match the behaviour of existing SQL engines (Postgres, Spark). Decimals should be first-class citizens.
### Additional context
**Related tickets:**
Function support:
- https://github.com/apache/datafusion/issues/17054
- https://github.com/apache/datafusion/issues/18031
- https://github.com/apache/datafusion/issues/17555
- https://github.com/apache/datafusion/issues/1545
- https://github.com/apache/datafusion/issues/7689
- https://github.com/apache/datafusion/issues/3572
Core support:
- https://github.com/apache/datafusion/issues/14612
- https://github.com/apache/datafusion/issues/18092
- https://github.com/apache/datafusion/issues/14760
- https://github.com/apache/datafusion/issues/14763
- https://github.com/apache/datafusion/issues/17489
- https://github.com/apache/datafusion/issues/17747
Coercion and type issues:
- https://github.com/apache/datafusion/issues/14272
- https://github.com/apache/datafusion/issues/8795
- https://github.com/apache/datafusion/issues/16667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Support Decimal for User Defined Functions [datafusion]
Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on issue #18889:
URL: https://github.com/apache/datafusion/issues/18889#issuecomment-3570439782
Thank you! I agree, some of them are too vague. We can move them to "nice-to-have" section or just omit. I will add my bunch of follow-up placeholder issues soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Support Decimal for User Defined Functions [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #18889:
URL: https://github.com/apache/datafusion/issues/18889#issuecomment-3570386481
Thanks for creating this; I'll be keen to help review & merge PRs relating to this epic 👍
I feel some of the listed issues might be better off being omitted from this epic to tighten it's focus, for example:
- https://github.com/apache/datafusion/issues/18092
- https://github.com/apache/datafusion/issues/14760
- https://github.com/apache/datafusion/issues/14763
These are more generic issues regarding signature usage so perhaps can leave them off?
- They are assigned to me, I've been meaning to work on them 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org