[GitHub] [arrow] drin commented on issue #13827: How to read a subset of columns from arrow ipc format file the fastest way?

[GitBox <gi...@apache.org>]

drin commented on issue #13827:
URL: https://github.com/apache/arrow/issues/13827#issuecomment-1209654088

   I think the absolute fastest way is to break up the columns into different files, in which case you'll have far fewer inefficiencies.
   
   I think structurally, IPC and parquet are similar, so performance should not drop when switching to IPC from parquet with regard to how much of a file must be read to select a subset of columns.
   
   I have ideas of experiments to run to make recommendations, but let me try and see if these happen to exist anywhere...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org