You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/04/24 16:57:38 UTC

[GitHub] [arrow-datafusion] alamb commented on issue #5076: Re-implement `DataFrame.write_*` to use `LogicalPlan::Write`

alamb commented on issue #5076:
URL: https://github.com/apache/arrow-datafusion/issues/5076#issuecomment-1520521168

   I think we should stay true to the design goal of DataFusion and keep this functionality as modular as possible (aka implemented in terms of traits that can be extended by other systems). 
   
   Here are some ideas:
   
   # Idea: Add a physical plan for LogicalPlan::DML
   (I think this is what @andygrove  is suggesting). 
   
    This would add a way to create a physical plan for `LogicalPlan::Dml(.. op: Insert)` and have that implementation call the  appropriate (TBD) methods on `TableProvider` that would handle writing. This is similar to what @metesynnada  proposes in https://github.com/apache/arrow-datafusion/pull/6049 though it is not mem table specific. 
   
   The upside here is we already have all the flow and planner and it would follow the pattern of system like spark (e.g.  [DataWritingCommandExec](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-SparkPlan-DataWritingCommandExec.html) -- thanks to @metesynnada  for the link)
   
   The downsides are that such an ExecutionPlan is kind of strange (it makes no output, so therefore most of the methods like "output ordering" are basically useless) as I mentioned on  https://github.com/apache/arrow-datafusion/pull/6049#pullrequestreview-1396922679
   
   # Idea: Add specific runner / executor for  Inserts / Update / Deletes 
   Maybe we could provide a function or struct  `run_insert(source: Arc<dyn ExecutionPlan>, target: Arc<dyn TableProvider>)` that would orchestrate:
   
   1. Running the execution plan
   2. calling appropriate (TBD) methods on `TableProvider` that would handle writing
   
   Here is how you might run it:
   
   ```rust
   let runner = Insert::new(context)
     .target(my_table)
     .run(target)?
   ```
   
   A  benefit here is that only systems that wanted to handle DML would invoke the inserter.
   
   A downside is that it would require more code / connections to work
   
   
   Maybe @avantgardnerio  has some thoughts in this area, as I think he has a system that does DML as well based on DataFusion
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org