[GitHub] [incubator-iceberg] danielcweeks commented on issue #9: Vectorize reads and deserialize to Arrow

[GitBox <gi...@apache.org>]

danielcweeks commented on issue #9: Vectorize reads and deserialize to Arrow
URL: https://github.com/apache/incubator-iceberg/issues/9#issuecomment-494886427
 
 
   For anyone interested, Spark already has a good reference implementation in Scala here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
   
   One thing to note is that there isn't an implementation for map type.  I think Arrow was planning on using list<struct<key,value>, but engines may need something closer to struct<keys:list<>, values:list<>> in order overlay the columnar implementations directly.  It might be necessary to provide both.
   
   I'm happy to help anyone who's interested in working on this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org