You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2025/12/11 07:09:08 UTC
[I] Implement FFI versions of runtime environment and execution properties [datafusion]
timsaucer opened a new issue, #19277:
URL: https://github.com/apache/datafusion/issues/19277
### Is your feature request related to a problem or challenge?
This is a follow on to https://github.com/apache/datafusion/pull/19223
To reduce the scope of that PR, two methods in the Session trait were not implemented, `runtime_env` and `execution_props`. It may be useful to some downstream users to have these available, specifically if any `TableProvider`s make use of their contents.
### Describe the solution you'd like
Follow the same design pattern as the rest of the FFI crate and implement these structs.
Expose them in `FFI_Session`
### Describe alternatives you've considered
We can leave as is for now, which is no worse than what currently exists before `FFI_Session` was implemented.
### Additional context
There has not been any user requests, so this may not be high priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement FFI versions of runtime environment and execution properties [datafusion]
Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on issue #19277:
URL: https://github.com/apache/datafusion/issues/19277#issuecomment-5949809952
Thanks for working on this. I just wanted to say that we are still interested in this. Datafusion is looking like the most reliably performant option we've tested, so we are keen to use it. However the FFI related issues when trying to use datafusion-python with [delta-rs](https://github.com/delta-io/delta-rs) is quite a big problem for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement FFI versions of runtime environment and execution properties [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #19277:
URL: https://github.com/apache/datafusion/issues/19277#issuecomment-5952755553
> Thanks for working on this. I just wanted to say that we are still interested in this. Datafusion is looking like the most reliably performant option we've tested, so we are keen to use it. However the FFI related issues when trying to use datafusion-python with [delta-rs](https://github.com/delta-io/delta-rs) is quite a big problem for us.
Can you share the specific problem you're having? Getting a runtime environment shared can be extremely complex, which is why it was left out. Do you have open issues you're running into? I'm happy to keep working on this but there's also a lot of other work we're looking to do so if we have specific problems it will help prioritize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement FFI versions of runtime environment and execution properties [datafusion]
Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on issue #19277:
URL: https://github.com/apache/datafusion/issues/19277#issuecomment-5954098954
The specific problem is https://github.com/delta-io/delta-rs/issues/4623. However I have just found a workaround using `deltalake.query.QueryBuilder`, so that significantly reduces the urgency from our perspective.
<details>
<summary>How we are working around it in-case anyone is interested</summary>
The limitation of this workaround is that we can only read delta tables.
We were always aiming to use `ibis` but the normal `datafusion` + `ibis` connection depends on `datafusion-python`. To read delta tables we need to use the delta table provider from `delta-rs`, which requires some complex objects to be moved over the FFI.
To avoid the FFI challenges we created `DeltaDataFusionBackend` which inherits from `ibis.backends.datafusion.Backend` and overrides a few methods. After compiling to SQL `DeltaDataFusionBackend` uses `deltalake.query.QueryBuilder` to execute it.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org