You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2024/07/29 15:50:26 UTC
[I] Allow for rust UDF extensions of datafusion-python [datafusion-python]
timsaucer opened a new issue, #792:
URL: https://github.com/apache/datafusion-python/issues/792
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Frequently when we are working on a set of new features or data analytics we find recurring patterns in our code. We will typically abstract these into a python package to share across the team. There are times where we must use UDFs to process some data that are non-performant since we have to convert the data to python objects when the existing pyarrow operations are insufficient for our needs.
I would like to be able to take these UDFs and author them in rust and use these user defined functions in python. Currently there is a blocker in that data cannot be shared from one rust crate to another. https://github.com/PyO3/pyo3/issues/1444 is a tracking issue that discusses this in more detail.
**Describe the solution you'd like**
It appears at first glance that `PyCapsule` approach is the best way forward based on examples in [pyo3_arrow](https://docs.rs/pyo3-arrow/latest/pyo3_arrow/) and [rust-numpy](https://github.com/PyO3/rust-numpy).
I have a minimal example that I will try to put up later this week into a repo to demonstrate what I would like to accomplish. Basically, I should be able to define a UDF in rust so that I can make it highly performant, add a `pyo3` wrapper function around it, and use this python module alongside the existing `datafusion-python`.
**Describe alternatives you've considered**
I'm uncertain at this point about other options besides a very painful reexport of the entire datafusion python module.
**Additional context**
I will try to push up my minimal demonstration repo later this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #792:
URL: https://github.com/apache/datafusion-python/issues/792#issuecomment-2271864716
If this PR goes in, we can possibly just update the user documentation with a minimal example and link to the blog post: https://github.com/apache/datafusion-site/pull/17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #792:
URL: https://github.com/apache/datafusion-python/issues/792#issuecomment-2260433609
https://github.com/timsaucer/tuple_filter_example/blob/main/src/lib.rs
I think my code must be making a copy of the data because it is slower than I would expect. I will continue to investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #792:
URL: https://github.com/apache/datafusion-python/issues/792#issuecomment-2260747603
Awesome! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #792: Document how to use rust UDF extensions of datafusion-python
URL: https://github.com/apache/datafusion-python/issues/792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #792:
URL: https://github.com/apache/datafusion-python/issues/792#issuecomment-2260366798
Based on excellent feedback on the discord server, I have a minimum demonstrable of doing this that does not require any code changes in datafusion-python. I'm going to document what I have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Document how to use rust UDF extensions of datafusion-python [datafusion-python]
Posted by "Michael-J-Ward (via GitHub)" <gi...@apache.org>.
Michael-J-Ward commented on issue #792:
URL: https://github.com/apache/datafusion-python/issues/792#issuecomment-2260736328
> I think my code must be making a copy of the data because it is slower than I would expect. I will continue to investigate.
Try using `maturin develop --release` in `tuple_filter_example`
Running with `maturin develop`
```
❯ python python-udf-comparisons.py
Simple filtering has number 32 rows and took 1.7959909439086914 s
This is the incorrect number of rows!
Explicit filtering has number 21 rows and took 0.9604465961456299 s
UDF filtering has number 21 rows and took 8.786761999130249 s
UDF filtering using pyarrow compute has number 21 rows and took 0.7557330131530762 s
UDF filtering using a custom rust function has number 21 rows and took 2.760791063308716 s
```
Re-running with `maturin devleop --release`
```
❯ python python-udf-comparisons.py
Simple filtering has number 32 rows and took 1.6849780082702637 s
This is the incorrect number of rows!
Explicit filtering has number 21 rows and took 0.9972596168518066 s
UDF filtering has number 21 rows and took 8.215271472930908 s
UDF filtering using pyarrow compute has number 21 rows and took 0.777594804763794 s
UDF filtering using a custom rust function has number 21 rows and took 0.6748173236846924 s
```
**NOTE**:
I had to convert all `int32` to `int64` types.
```
❯ python python-udf-comparisons.py
Simple filtering has number 32 rows and took 1.810662031173706 s
This is the incorrect number of rows!
Explicit filtering has number 21 rows and took 0.9524128437042236 s
UDF filtering has number 21 rows and took 8.34911322593689 s
Traceback (most recent call last):
File "/home/mike/workspace/tuple_filter_example/python-example/python-udf-comparisons.py", line 185, in <module>
num_rows = df_udf_pyarrow_compute.count()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/workspace/datafusion-python/dev/python/datafusion/dataframe.py", line 508, in count
return self.df.count()
^^^^^^^^^^^^^^^
Exception: type_coercion
caused by
Error during planning: Coercion from [Int64, Int64, Utf8] to the signature Exact([Int32, Int32, Utf8]) failed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org