You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/01/06 20:36:09 UTC

[PR] Document the ParquetRecordBatchStream buffering [arrow-rs]

alamb opened a new pull request, #6947:
URL: https://github.com/apache/arrow-rs/pull/6947

   # Which issue does this PR close?
   
   - Related to https://github.com/apache/arrow-rs/issues/6946
   
   # Rationale for this change
   There was some surprise that the `ParquetRecordBatchStream` buffered entire row groups during decode, so let's documet this better.
   
   - Here is a proposal /  discussion on changing the strategy: https://github.com/apache/arrow-rs/issues/6946
   
   
   # What changes are included in this PR?
   1. Document buffering in `ParquetRecordBatchStream`
   
   # Are there any user-facing changes?
   Docs
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please call them out.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Document the `ParquetRecordBatchStream` buffering [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6947:
URL: https://github.com/apache/arrow-rs/pull/6947#discussion_r1905188237


##########
parquet/src/arrow/async_reader/mod.rs:
##########
@@ -611,11 +611,22 @@ impl<T> std::fmt::Debug for StreamState<T> {
     }
 }
 
-/// An asynchronous [`Stream`](https://docs.rs/futures/latest/futures/stream/trait.Stream.html) of [`RecordBatch`]
-/// for a parquet file that can be constructed using [`ParquetRecordBatchStreamBuilder`].
+/// An asynchronous [`Stream`]of [`RecordBatch`] constructed using [`ParquetRecordBatchStreamBuilder`] to read parquet files.
 ///
 /// `ParquetRecordBatchStream` also provides [`ParquetRecordBatchStream::next_row_group`] for fetching row groups,
 /// allowing users to decode record batches separately from I/O.
+///
+/// # I/O Buffering
+///
+/// `ParquetRecordBatchStream` buffers *all* data pages selected after predicates
+/// (projection + filtering, etc) and decodes the rows from those buffered pages.
+///
+/// For example, if all rows and columns are selected, the entire row group is
+/// buffered in memory during decode. This minimized the number of IO operations
+/// required.

Review Comment:
   ```suggestion
   /// buffered in memory during decode. This minimizes the number of IO operations
   /// required, which is especially important for object stores, where IO operations
   /// have latencies in the hundreds of milliseconds
   ```
   
   I think it is important to provide the why as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Document the `ParquetRecordBatchStream` buffering [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6947:
URL: https://github.com/apache/arrow-rs/pull/6947#issuecomment-2577748075

   Thanks @tustvold 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Document the `ParquetRecordBatchStream` buffering [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6947:
URL: https://github.com/apache/arrow-rs/pull/6947


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org