[GitHub] [incubator-iceberg] samarthjain opened a new issue #647: Prefetch parquet data pages

[GitBox <gi...@apache.org>]

samarthjain opened a new issue #647: Prefetch parquet data pages
URL: https://github.com/apache/incubator-iceberg/issues/647
 
 
   While doing a hot spot analysis of the benchmark runs we have for vectorized read path, reading and decompressing of the parquet data pages stood out as one of the major hot spots. 
   
   We can significantly improve performance if we can take out this read/decompress from the hot code path by prefetching the next data page asynchronously. Preliminary benchmark runs showed up to 60% performance gains when using prefetching. 
   An initial implementation is here - https://github.com/samarthjain/incubator-iceberg/tree/samarth_prefetch_vectorized
   
   This is going to need some Parquet level changes as the underlying native compressors used are not thread safe. https://github.com/apache/parquet-mr/pull/671 adds support for airlift based compressors which are written in pure Java and don't use any JNI resources.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org