You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/11/03 15:44:56 UTC

[I] [Parquet] Pre-fetch the next row group when reading parquet files [datafusion]

alamb opened a new issue, #18470:
URL: https://github.com/apache/datafusion/issues/18470

   ### Is your feature request related to a problem or challenge?
   
   The current Parquet reader has the following pattern:
   1. Do I/O to buffer data needed for row groups
   2. Decode the data (make RecordBatches)
   3. fetching IO for next row group
   4. ...
   
   The issue is that this means that after the row group is finished, the CPU will then have to wait for IO before it can begin decoding the next row group. We could use the CPUs more efficiently if we overlapped IO and CPU more
   
   
   ### Describe the solution you'd like
   
   Prefetch the next row group (to start the IO) while actively reading the current group
   
   ### Describe alternatives you've considered
   
   I made a POC of this approach and it looks promising (improves performance locally for very short queries)
   - https://github.com/apache/datafusion/pull/18391
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Parquet] Pre-fetch the next row group when reading parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18470:
URL: https://github.com/apache/datafusion/issues/18470#issuecomment-3863046575

   Here is another (self serving) paper that discusses the use of caches: [The Five-Minute Rule for the Cloud: Caching in Analytics Systems](https://vldb.org/cidrdb/papers/2025/p4-duwe.pdf)
   
   > If we have the prefetch, we can reduce the dependency of the cache layer and save more resources.
   
   In general I agree this is true
   
   The challenge I worry about about is that if you aren't careful, prefetching results in buffering more than necessary (aka fetch faster than the CPU can consume it) thus using more memory than needed
   
   So the optimal prefetching strategy depends heavily on what your IO looks like -- for example the pattern is quite different from object store and "local disk" -- and even pretty different if local disk is an actual SSD or a cloud "EBS" like block store. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Parquet] Pre-fetch the next row group when reading parquet files [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #18470:
URL: https://github.com/apache/datafusion/issues/18470#issuecomment-3858372657

   I thought about the prefetch last week while reading the [paper](https://research.google/pubs/napa-powering-scalable-data-warehousing-with-robust-query-performance-at-google/).
   
   Currently, most databases adopt the storage-computation disaggregation architecture, most of them read from object storage +tranditional disk cache. The cache seems to be necessary for the latency-sensitive business because of the unpredictable object store long-tail latency. 
   
   If we have the prefetch, we can reduce the dependency of the cache layer and save more resources.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Parquet] Pre-fetch the next row group when reading parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18470:
URL: https://github.com/apache/datafusion/issues/18470#issuecomment-3863050132

   (the memory tradeoff BTW is why I haven't pursued https://github.com/apache/datafusion/pull/18391 more)
   
   Some other ideas
   1. Provide APIs for users of DataFusion to more easily hook up their own caches
   2. Make some sort of adaptive I/O thing
   
   Both of those options sound complicated to me, though


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org