[GitHub] [arrow-datafusion] yjshen edited a comment on issue #1916: Discussion: Is Ballista a standalone system or framework

[GitBox <gi...@apache.org>]

yjshen edited a comment on issue #1916:
URL: https://github.com/apache/arrow-datafusion/issues/1916#issuecomment-1058889293


   > I think there is another side of this question- do we consider DataFusion as a pure library? Would this change how things are organized? If so, what is the API to interface with other clients, logical plan? physical plan? And where would SQL parsing, optimizer and planner go?
   
   Yes, I think so. And actually, we are already heading in that direction. The first step to make each module substitutable is to make each part, SQL parsing, optimizing, and executing in its own crate. @Jimexist is leading the effort in https://github.com/apache/arrow-datafusion/issues/1750. Once this separating of functionalities is done, the next step would be introducing interfaces between adjacent functionalities, depending on how would we substitute modules. 
   
   One of the possibilities I'm sure is, once we have https://github.com/apache/arrow-datafusion/pull/1887 or even https://github.com/andygrove/substrait-rs, we can link Apache Calcite to DataFusion's physical plan execution layer.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org