[GitHub] [incubator-iceberg] prodeezy commented on issue #9: Vectorize reads and deserialize to Arrow

[GitBox <gi...@apache.org>]

prodeezy commented on issue #9: Vectorize reads and deserialize to Arrow
URL: https://github.com/apache/incubator-iceberg/issues/9#issuecomment-517486998
 
 
   After running benchmarks met with @samarthjain , @anjalinorwood and @rdblue  to go over some possible improvements : 
   
   
   Vectorization Perf Meeting notes (Aug 1):
   
   Possible low hanging fruit to reap with perf : 
     - Remove virtual calls in vector reader, make concrete classes (Anjali)
     - Remove hasNext() calls and currentDefinitionLevels() checks and tighten the VectorReader.read() loop (Anjali)
     - Look into arrow memory allocation, try to preallocate memory at initialization and avoid re-allocations ( Samarth ??)
     - Slowness seen in time spent in FieldVector.setSafe() calls Improve Decimal / String variable byte reading. Make this fixed length based or using binary reading (Gautam)
   
   Deeper look :
     - Possibly look into removing advance() calls at every stage in ColumnIterator
     - Look into removing triple iterator abstraction and work at lower level with parquet 
   
   Lemme know if this list looks ok to you guys and if i missed anything. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org