[GitHub] [arrow-rs] alamb commented on issue #158: Parquet ArrayReader should allow reading a subset of row groups

[GitBox <gi...@apache.org>]

alamb commented on issue #158:
URL: https://github.com/apache/arrow-rs/issues/158#issuecomment-826804565


   Comment from Andy Grove(andygrove) @ 2020-12-23T16:28:31.604+0000:
   <pre>[~nevi_me] [~sunchao] Do either of you know if this would be a lot of work to implement or not? If you have any pointers for someone working on this ticket it would be appreciated.</pre>
   
   Comment from Neville Dipale(nevi_me) @ 2020-12-24T19:37:44.895+0000:
   <pre>I'm not yet sure about what it would take to read the same file across threads. Would that be like sharing the file handle, as that might not work on all OSes.
   
    
   
   From working with the arrow -> parquet IO, I think that if the batch size is aligned with record group size, it might make it easier to read partial data from 1 or more groups.
   
   I'll have a look in the coming days, and update this ticket with more info. It might likely be a case of exposing some method to do partial reads.</pre>
   
   Comment from Chao Sun(csun) @ 2020-12-29T01:44:48.654+0000:
   <pre>Sorry for the late reply. Yes I think it should be possible. On the file reader side we can pass in a (start, end) besides the file handle, to indicate we want to only read a segment of the file. Then after parsing the file metadata, we can check all the row groups for the file and determine which row group(s) overlaps with the segment, and only select those. 
   
   You can probably check relevant code in [Spark|https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L105] and [Parquet|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1223] for reference.
   
   I'm not sure about the file handle sharing issue [~nevi_me] mentioned tho - I thought we used to clone file handle so that they can be shared but yeah haven't looked at the code base for some time :(</pre>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org