You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2025/02/09 15:05:49 UTC
[I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
timsaucer opened a new issue, #1017:
URL: https://github.com/apache/datafusion-python/issues/1017
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Suppose someone wants to build a library that is usable by both rust and python DataFusion users. They have written a UDF in rust and it implements the rust DataFusion traits (whether scalar, aggregate, or window). Right now, if that user wants to use their UDF in `datafusion-python`, they need to expose a variety of methods that basically mimic the trait functions of the rust code. For scalar UDFs the interface requires a bit of wrangling from ColumnarValue to PyArrow objects.
While it is possible to do this, it is likely error prone and tedious for implementers to write and maintain this code.
**Describe the solution you'd like**
We have an established pattern of adding foreign table providers via FFI interface and using PyCapsule. This makes adding a TableProvider a very easy operation. In our example code, the function to expose a table provider is only 6 lines of code and likely will require minimal maintenance.
It would be nice to expose all of the varieties of user defined functions via FFI to make this follow the established pattern and also easy for users to reuse their code.
**Describe alternatives you've considered**
I did a brief proof of concept where I used python calls to the required functions. This did work, but it took quite a bit of code and I suspect it will be difficult to maintain.
**Additional context**
This may provide additional value in that it would get us much closer to being able to expose a `SessionContext` via ffi, which would have nice impacts to both the datafusion-ray and ballista projects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1017:
URL: https://github.com/apache/datafusion-python/issues/1017#issuecomment-2646550034
@Spaarsh I’m thinking the latter. I should have a draft PR for the scalar variant in the next few days to demonstrate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
Posted by "Spaarsh (via GitHub)" <gi...@apache.org>.
Spaarsh commented on issue #1017:
URL: https://github.com/apache/datafusion-python/issues/1017#issuecomment-2646557713
@timsaucer okay! If no one is doing this already, I will try and understand how it can be done for Aggregator or Window UDFs. Though I am unclear of the approach, I am going through the TableProvider UDF code to gain some clarity! Any tips are helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1017:
URL: https://github.com/apache/datafusion-python/issues/1017#issuecomment-2647858283
@Spaarsh I've put up a *draft* PR for the scalar udf, but it has a few points that need cleaning up still: https://github.com/apache/datafusion/pull/14579
One thing to be wary of when going down this direction is that it is very important to clearly understand which side of the FFI code exists in the provider and which is the consumer. I've tried to follow a pattern of `FFI_` being the provider and `Foreign*` being the consumers. Additionally we want to be careful about how much of the DataFusion API we want and need to expose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
Posted by "Spaarsh (via GitHub)" <gi...@apache.org>.
Spaarsh commented on issue #1017:
URL: https://github.com/apache/datafusion-python/issues/1017#issuecomment-2646548490
@timsaucer so if I understood this correctly, we are trying to either create an ```FFI_UDF``` that is capable of ingesting different types of UDF or, we create an FFI for each kind of UDF i.e., ```FFI_UDF_Scalar```, ```FFI_UDF_Window``` and so on. Right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to use rust DataFusion UDFs in datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #1017: Make it easier to use rust DataFusion UDFs in datafusion-python
URL: https://github.com/apache/datafusion-python/issues/1017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org