[I] ZeroCopy Conversion from Spark ColumnarBatch [datafusion-comet]

["tokoko (via GitHub)" <gi...@apache.org>]

tokoko opened a new issue, #3518:
URL: https://github.com/apache/datafusion-comet/issues/3518

   ### What is the problem the feature request solves?
   
   When spark DataSource api can read data as ColumnarBatch objects datafusion can already do a direct columnar to columnar conversion. This avoids row conversion, but still means data needs to be copied even when the data in ColumnarBatch is already arrow.
   
   ### Describe the potential solution
   
   Conversion should first check if vectors in ColumnarBatch are instances of `org.apache.spark.sql.vectorized.ArrowColumnVector`. When that's true, conversion should be zero-copy and probably ignore datafusion's maxRowsPerBatch config (??).
   
   ### Additional context
   
   I'm working on [spark-adbc](https://github.com/tokoko/spark-adbc) data source. data source outputs `ArrowColumnVector`s in ColumnarBatch. while using comet helps avoid row conversion right after read, there's still some penalty for copying vectors over.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org