You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ryansun96 (via GitHub)" <gi...@apache.org> on 2026/01/28 01:12:44 UTC

[I] Make PyDataFrame.inner_df() public [datafusion-python]

ryansun96 opened a new issue, #1354:
URL: https://github.com/apache/datafusion-python/issues/1354

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   Current python bindings allow users to implement rust extension while majority of their logic stays in python. In my project, I'm trying to do the opposite: having majority of the logic in rust with some entry point in Python. The user would make a DataFrame from some raw data in Python and pass it to the rust logic for processing.
   
   Current implementation of `PyDataFrame` does not allow the "real / inner" rust data frame to be accessed, because `inner_df` is `pub(crate)`.
   
   **Describe the solution you'd like**
   
   Make `PyDataFrame.inner_df()` `pub`.
   
   **Describe alternatives you've considered**
   
   Open to advice if the above workflow is not recommended, as I'm quite new to datafusion and rust.
   
   **Additional context**
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3811426596

   It sounds like your final use case will not have `datafusion-python` in the loop as a **Python dependency** at all, correct?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-4216042134

   Does this continue to be an issue? Is there anything else you need help with?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3811627310

   Approach 1: Using df-python
   - Write all of your rust code for UDFs, etc and expose via PyO3 using the FFI approach.
   - Register these with df-python
   - Write python style dataframe operations
   - Everything still runs at full native speed during execution with zero copy between UDFs and core DF code
   
   Approach 2: Using df-python as a **rust** dependency only
   - We update the method to `pub` as requested
   - You expose all of the specific endpoints you want to use to python, so probably something that is taking in that pandas dataframe in your method signature
   - Everything is running in Rust and the results are sent back probably as arrow record batch or record batch stream
   - You will not get any of the datafusion-python python API (but maybe not an issue)
   
   Approach 3: Mixture of the two
   - Write a table provider that takes in as an input that pandas dataframe, does all the operations you want under the hood using its own session context (not the datafusion-python session context)
   - Follow on operations from the output of the table provider could use datafusion-python interfaces


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-4221362829

   Hi @timsaucer thanks for checking in! I decided to pass arrow RecordBatches from Python to Rust based on your suggestion above, so we do not need to make any changes to df-py any more. Thanks a lot!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3817528113

   https://github.com/apache/arrow-rs/blob/caf1c6022c71af00ef712e9e865acfee74169f0d/arrow-pyarrow/src/lib.rs#L461 shows how you can get arrow stream from Python, which can then be fed into a table provider in DataFusion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 closed issue #1354: Make PyDataFrame.inner_df() public
URL: https://github.com/apache/datafusion-python/issues/1354


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3814642156

   > final use case will not have datafusion-python in the loop as a **Python** dependency at all, correct?
   
   That is correct. One detail that I did not mention in my previous post but seems to be significant based on the proposed approaches, is that in our project, we do not need to send the rust processing results back to Python, which makes Approach 2 much simpler.
   
   With that I do think Approach 3 has value outside of our use case right now. To check my understanding, approach 3 is essentially using a table provider as a "pipe" operator that embeds custom logic, rather than simply reading & returning rows?
   
   > does all the operations you want under the hood using its own session context (not the datafusion-python session context)
   
   Does it imply the rule of thumb that "If I want to use functions from different libraries in the same session context, those libraries need to be compiled together"? I.e. even if use datafusion-python as a **py** dependency, as long as we depend on source, not the artifact published to PyPI, we will be fine even if we do data transfer "naively"?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3808523359

   I recommend reading this section of the online manual, specifically the section about the unstable ABI: https://datafusion.apache.org/python/contributor-guide/ffi.html
   
   This is the reason we do not wish to make`inner_df` `pub`. It is *not* recommended to use datafusion-python as a direct dependency. Rather, we wish users to use `datafusion` dependencies and work through FFI wherever possible.
   
   This is an area of active development! We'd love to have additional people get involved.
   
   Can you give us a little more information about what specifically you're trying to do? We may already have work underway that can help you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3808653833

   Hi @timsaucer thanks for your reply! Happy to share more context:
   
   We are rewriting a Python system in rust, in a phased approach. In simplified terms, our current system looks like this:
   
   ```python
   class BaseCls:
       def method1(df: pd.DataFrame):
           pass #Some pandas logic
       def method2():
           pass
   ```
   To make the change as transparent as possible, in Phase 1, we plan to implement certain compute and memory intensive operations in rust, using datafusion, exposed via PyO3 bindings. In other words, what we are trying to do is:
   ```python
   class BaseCls:
       def method1(df: pd.DataFrame): #Now a native method implemented in Rust, using datafusion for various data frame operations
           
       def method2():
           pass # Same as before, implemented in Py
   ```
   
   Thus, we would like to access & manipulate the dataframe from rust, rather than using the exposed py bindings. We will implement rust UDFs, object store registries, etc., without using those items from py at all.
   
   What is your recommended approach for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3814646797

   Would it be cool for me to open a PR to implement Approach 2, update the method to pub?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Make PyDataFrame.inner_df() public [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1354:
URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3817494812

   I'm not opposed to the PR, but have you verified locally it actually solves your problem? Also if you aren't sending anything back to Python and you're not starting with a datafusion dataframe (or are you?) then maybe all you really need is a method to get arrow record batches (streaming or otherwise) into datafusion core. You could potentially lift the appropriate snippet and reduce your compile times significantly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org