[GitHub] [arrow] andygrove edited a comment on pull request #8870: ARROW-10854: [Rust] [DataFusion] Simplify logical plan scans

[GitBox <gi...@apache.org>]

andygrove edited a comment on pull request #8870:
URL: https://github.com/apache/arrow/pull/8870#issuecomment-741855709


   I think it would be helpful if I explained my design philosophy on this but also bear in mind that I am not an expert on this but this was my initial thinking:
   
   - Logical plans should be trivially serializable. This is a good forcing function to make sure that they can easily be constructed from any programming language and don't become Rust-specific. This is important when building bindings in other languages such as Python and Java and also for building drivers for distributed systems so that we can serialize query plans and send them to other processes.
   - The execution context has registries of things like tables and udfs and is responsible for translating a logical plan into a physical plan, and it makes sense for the physical plan to use opaque dyn traits.
   
   In Ballista, I have my own fork of the DataFusion logical plan currently, and I could continue with that but was considering using the DataFusion plan directly at some point (I'm not actively working on Ballista at the moment so this is all a bit theoretical).
   
   I bet @alamb will have some valuable opinions on this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org