You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EmilyMatt (via GitHub)" <gi...@apache.org> on 2026/09/16 10:18:24 UTC
[PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
EmilyMatt opened a new pull request, #25372:
URL: https://github.com/apache/datafusion/pull/25372
Currently the behaviour is to only keep the current batch reserved per cursor, meaning we don't *actually* reserve all the rows we are keeping owned.
This leads to a lot of invisible memory, that is held, but not accounted by the pool.
(This scales fairly linearly with the number of partitions, in a regular round-robin case, for example.)
While the mem overhead was greatly reduced by @ariel-miculas 's https://github.com/apache/datafusion/pull/23619 and https://github.com/apache/datafusion/pull/23802
It is still a very problematic area in terms of memory visibility.
(Round-robin is an extreme case by nature)
This PR does *not* reduce the mem usage or anything, it just makes the memory *visible* and keeps everything that needs to be reserved - reserved.
I believe in order to ensure robustness, once this is merged another PR will need to be made to cap the merge reservation to the peak of the ExternalSorter reservation (due to some behaviours I've seen with fair-type pools, which treat spillable and non-spillable operators differently)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Account for buffered rows in `RowCursorStream` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #25372:
URL: https://github.com/apache/datafusion/pull/25372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4120771802
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -94,25 +94,62 @@ impl FusedStreams {
}
}
-/// An `Arc<Rows>` that can be reused
+/// An `Arc<Rows>` that can be reused.
+///
+/// Owns the reservation covering every retained buffer for as long as it is
+/// retained, so the cache is visible to the pool rather than held off-book.
+/// A retained buffer keeps its capacity, so the cost is the high-water mark of
+/// each stream, not the size of the batch currently in flight.
#[derive(Debug)]
struct ReusableRows {
inner: Vec<Option<Arc<Rows>>>,
+ reservation: MemoryReservation,
}
impl ReusableRows {
// return a Rows for writing,
// does not clone if the existing rows can be reused
- fn take_next(&mut self, stream_idx: usize) -> Result<Rows> {
- Arc::try_unwrap(self.inner[stream_idx].take().unwrap()).map_err(|_| {
- internal_datafusion_err!(
- "Rows from RowCursorStream is still in use by consumer"
- )
- })
+ fn take_next(&mut self, stream_idx: usize, converter: &RowConverter) -> Result<Rows> {
+ match self.inner[stream_idx].take() {
+ Some(rows) => Arc::try_unwrap(rows).map_err(|_| {
+ internal_datafusion_err!(
+ "Rows from RowCursorStream is still in use by consumer"
+ )
+ }),
+ // Nothing retained yet, or already released, so start over.
+ None => Ok(converter.empty_rows(0, 0)),
+ }
}
- // save the Rows
- fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) {
+
+ /// Account for a freshly built buffer, and retain it for reuse.
+ ///
+ /// The reservation is mandatory rather than best-effort. The buffer is live in the
+ /// cursor whether or not this slot keeps a handle to it, so declining to reserve
+ /// would hide it from the pool instead of avoiding it, and it frees nothing at this
+ /// point either, since the cursor holds the same `Arc`. Retention on top of the
+ /// reservation is free for the same reason. A pool that cannot cover the buffer
+ /// fails the query here, as it did before the buffer was cached at all.
+ fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) -> Result<()> {
self.inner[stream_idx] = Some(Arc::clone(rows));
+ let retained = self.retained_size();
+ debug_assert!(retained >= self.reservation.size());
+ if let Err(e) = self.reservation.try_resize(retained) {
Review Comment:
You're right, addressed, though am now in a bit of a philosophical predicament on which error to return, the original try_resize or the supposedly "shrinking" try_resize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Account for buffered rows in `RowCursorStream` [datafusion]
Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#issuecomment-6192538006
I found a performance follow-up: recalculating retained-buffer memory scans every merge input for each batch. #26366 makes these updates constant-time while preserving the reservations. The included SQL benchmark improves from 4.82 s to 1.30 s locally.
@EmilyMatt and @andygrove, since you're obviously familiar with this part of the code, would you mind taking a look? 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4271166933
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -587,4 +635,211 @@ mod tests {
Ok(())
}
+
+ /// Drives a `RowCursorStream` over several partitions and reports the pool
+ /// reservation, so the tests below can watch it move.
+ struct RowCursorHarness {
+ stream: RowCursorStream,
+ pool: Arc<dyn MemoryPool>,
+ schema: SchemaRef,
+ rows: usize,
+ str_width: usize,
+ }
+
+ /// Every batch has the same row count and string width, so each converts to
+ /// a `Rows` buffer of the same size regardless of `seq`.
+ fn make_batch(
+ schema: &SchemaRef,
+ seq: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> RecordBatch {
+ let base = (seq * rows) as i32;
+ let a = Int32Array::from_iter_values((0..rows as i32).map(|i| base + i));
+ let b = StringArray::from_iter_values((0..rows).map(|_| "x".repeat(str_width)));
+ RecordBatch::try_new(Arc::clone(schema), vec![Arc::new(a), Arc::new(b)]).unwrap()
+ }
+
+ impl RowCursorHarness {
+ /// `batches_per_partition` batches of `rows` rows each, per partition.
+ /// The sort key spans two columns so this takes the `Rows` path rather
+ /// than the specialized single-column `FieldCursorStream`.
+ fn new(
+ partitions: usize,
+ batches_per_partition: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> Result<Self> {
+ let schema: SchemaRef = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Int32, false),
+ Field::new("b", DataType::Utf8, false),
+ ]));
+
+ let streams: Vec<SendableRecordBatchStream> = (0..partitions)
+ .map(|_| {
+ let batches: Vec<RecordBatch> = (0..batches_per_partition)
+ .map(|seq| make_batch(&schema, seq, rows, str_width))
+ .collect();
+ Box::pin(
+ MemoryStream::try_new(batches, Arc::clone(&schema), None)
+ .unwrap(),
+ ) as SendableRecordBatchStream
+ })
+ .collect();
+
+ let expressions = LexOrdering::new(vec![
+ PhysicalSortExpr::new_default(col("a", &schema)?),
+ PhysicalSortExpr::new_default(col("b", &schema)?),
+ ])
+ .unwrap();
+
+ // Generous limit: this measures what is reserved, not what is refused.
+ let pool: Arc<dyn MemoryPool> =
+ Arc::new(GreedyMemoryPool::new(256 * 1024 * 1024));
+ let reservation = MemoryConsumer::new("test").register(&pool);
+ let stream =
+ RowCursorStream::try_new(&schema, &expressions, streams, reservation)?;
+
+ Ok(Self {
+ stream,
+ pool,
+ schema,
+ rows,
+ str_width,
+ })
+ }
+
+ fn reserved(&self) -> usize {
+ self.pool.reserved()
+ }
+
+ /// Converts one batch with a standalone converter over the same sort
+ /// key, returning `(converter size, size of one retained Rows buffer)`.
+ /// These are what the stream should reserve for its converter and for
+ /// each partition's cached buffer.
+ fn expected_sizes(&self) -> Result<(usize, usize)> {
+ let converter = RowConverter::new(vec![
+ SortField::new(DataType::Int32),
+ SortField::new(DataType::Utf8),
+ ])?;
+ let batch = make_batch(&self.schema, 0, self.rows, self.str_width);
+ let mut rows = converter.empty_rows(0, 0);
+ converter.append(&mut rows, batch.columns())?;
+ Ok((converter.size(), rows.size()))
+ }
+
+ /// Polls `stream_idx`, handing back the cursor so the caller controls when
+ /// it is dropped. `None` once the partition is finished.
+ fn poll_cursor(&mut self, stream_idx: usize) -> Result<Option<RowValues>> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => Ok(Some(values)),
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(None),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+
+ /// Polls `stream_idx` and immediately drops the returned cursor, which is
+ /// what the merge does once a cursor is exhausted. Returns `false` once
+ /// the partition is finished.
+ fn poll_and_drop_cursor(&mut self, stream_idx: usize) -> Result<bool> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => {
+ drop(values);
+ Ok(true)
+ }
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(false),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+ }
+
+ // `RowCursorStream` keeps one `Rows` buffer per partition alive between
+ // polls so the allocation can be reused. That buffer is real memory, and it
+ // outlives the cursor handed to the consumer: the merge drops a cursor as
+ // soon as it is exhausted, but the buffer behind it stays cached.
+ //
+ // So dropping the cursor must not change what the pool reports. If it does,
+ // the bytes still held by the cache have gone off the books and the pool is
+ // under-counting a live allocation.
+ #[test]
+ fn dropping_a_cursor_does_not_unaccount_its_retained_buffer() -> Result<()> {
+ let mut harness = RowCursorHarness::new(1, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ assert_eq!(harness.reserved(), 0);
+
+ let cursor = harness.poll_cursor(0)?.expect("first batch");
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ // The cached `Rows` buffer outlives the cursor, so dropping the cursor
+ // must not release its bytes.
+ drop(cursor);
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ Ok(())
+ }
+
+ // The retained bytes are per partition, so what the pool reports has to
+ // scale with partition count rather than staying flat at the size of
+ // whichever batch happens to be in flight.
+ #[test]
+ fn retained_row_buffer_reservation_scales_with_partitions() -> Result<()> {
+ for partitions in [2, 16] {
+ let mut harness = RowCursorHarness::new(partitions, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ for stream_idx in 0..partitions {
+ assert!(harness.poll_and_drop_cursor(stream_idx)?);
+ }
+ // One shared converter, plus one cached buffer per partition.
+ assert_eq!(
+ harness.reserved(),
+ converter_size + partitions * rows_size,
+ "{partitions} partitions"
+ );
+ }
+
+ Ok(())
+ }
+
+ // The flip side: a partition that will never be polled again must hand its
+ // buffer back. Otherwise the reservation only ever grows and a long merge
+ // holds every partition's high-water mark until the whole stream drops.
+ #[test]
+ fn exhausted_partitions_release_their_retained_buffers() -> Result<()> {
Review Comment:
Syned to main while I'm at it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#issuecomment-5696916106
> Currently the behaviour is to only keep the current batch reserved per cursor, meaning we don't actually reserve all the rows we are keeping owned.
This leads to a lot of invisible memory, that is held, but not accounted by the pool.
I think this should be explained in more detail.
Specifically, from what I can tell, the existing issue is that we release the memory reservation when the cursor for a partition is dropped, ignoring the fact that ReusableRows keeps the memory alive.
Other than that, the [cursors Vec](https://github.com/apache/datafusion/blob/0646a310cdeb25ba6a091e643dddd06da7ea181e/datafusion/physical-plan/src/sorts/merge.rs#L90) should keep the memory reservations alive for each input partition, and when the stream is polled, the previous reservation would be dropped and a new reservation created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#issuecomment-5696202666
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25372?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `82.06897%` with `26 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.94%. Comparing base ([`86ba5e0`](https://app.codecov.io/gh/apache/datafusion/commit/86ba5e05fccc3e9e79f6d19ee5888c46fbb31117?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9b992dd`](https://app.codecov.io/gh/apache/datafusion/commit/9b992dd5c4a1649d3c7af6eedcf9599172480dcf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25372?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/sorts/stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/25372?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fsorts%2Fstream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9zb3J0cy9zdHJlYW0ucnM=) | 81.81% | [9 Missing and 17 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25372 +/- ##
==========================================
- Coverage 81.94% 81.94% -0.01%
==========================================
Files 1135 1135
Lines 428278 428403 +125
Branches 428278 428403 +125
==========================================
+ Hits 350947 351049 +102
- Misses 56375 56383 +8
- Partials 20956 20971 +15
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25372?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4120749747
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -94,25 +94,62 @@ impl FusedStreams {
}
}
-/// An `Arc<Rows>` that can be reused
+/// An `Arc<Rows>` that can be reused.
+///
+/// Owns the reservation covering every retained buffer for as long as it is
+/// retained, so the cache is visible to the pool rather than held off-book.
+/// A retained buffer keeps its capacity, so the cost is the high-water mark of
+/// each stream, not the size of the batch currently in flight.
#[derive(Debug)]
struct ReusableRows {
inner: Vec<Option<Arc<Rows>>>,
+ reservation: MemoryReservation,
}
impl ReusableRows {
// return a Rows for writing,
// does not clone if the existing rows can be reused
- fn take_next(&mut self, stream_idx: usize) -> Result<Rows> {
- Arc::try_unwrap(self.inner[stream_idx].take().unwrap()).map_err(|_| {
- internal_datafusion_err!(
- "Rows from RowCursorStream is still in use by consumer"
- )
- })
+ fn take_next(&mut self, stream_idx: usize, converter: &RowConverter) -> Result<Rows> {
+ match self.inner[stream_idx].take() {
+ Some(rows) => Arc::try_unwrap(rows).map_err(|_| {
+ internal_datafusion_err!(
+ "Rows from RowCursorStream is still in use by consumer"
+ )
+ }),
+ // Nothing retained yet, or already released, so start over.
+ None => Ok(converter.empty_rows(0, 0)),
+ }
}
- // save the Rows
- fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) {
+
+ /// Account for a freshly built buffer, and retain it for reuse.
+ ///
+ /// The reservation is mandatory rather than best-effort. The buffer is live in the
+ /// cursor whether or not this slot keeps a handle to it, so declining to reserve
+ /// would hide it from the pool instead of avoiding it, and it frees nothing at this
+ /// point either, since the cursor holds the same `Arc`. Retention on top of the
+ /// reservation is free for the same reason. A pool that cannot cover the buffer
+ /// fails the query here, as it did before the buffer was cached at all.
+ fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) -> Result<()> {
self.inner[stream_idx] = Some(Arc::clone(rows));
+ let retained = self.retained_size();
+ debug_assert!(retained >= self.reservation.size());
+ if let Err(e) = self.reservation.try_resize(retained) {
+ self.inner[stream_idx] = None;
+ return Err(e);
+ }
+ Ok(())
+ }
+
+ // drop whatever a finished stream was holding
+ fn release(&mut self, stream_idx: usize) {
+ debug_assert!(self.reservation.size() >= self.retained_size());
+ if let Some(rows) = self.inner[stream_idx].take() {
+ self.reservation.shrink(rows.size());
+ }
+ }
+
+ fn retained_size(&self) -> usize {
Review Comment:
Yeah it's more of a design choice, I think it's negligible performance-wise, and makes for a cleaner, more stateless calculation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4116874032
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -94,25 +94,62 @@ impl FusedStreams {
}
}
-/// An `Arc<Rows>` that can be reused
+/// An `Arc<Rows>` that can be reused.
+///
+/// Owns the reservation covering every retained buffer for as long as it is
+/// retained, so the cache is visible to the pool rather than held off-book.
+/// A retained buffer keeps its capacity, so the cost is the high-water mark of
+/// each stream, not the size of the batch currently in flight.
#[derive(Debug)]
struct ReusableRows {
inner: Vec<Option<Arc<Rows>>>,
+ reservation: MemoryReservation,
}
impl ReusableRows {
// return a Rows for writing,
// does not clone if the existing rows can be reused
- fn take_next(&mut self, stream_idx: usize) -> Result<Rows> {
- Arc::try_unwrap(self.inner[stream_idx].take().unwrap()).map_err(|_| {
- internal_datafusion_err!(
- "Rows from RowCursorStream is still in use by consumer"
- )
- })
+ fn take_next(&mut self, stream_idx: usize, converter: &RowConverter) -> Result<Rows> {
+ match self.inner[stream_idx].take() {
+ Some(rows) => Arc::try_unwrap(rows).map_err(|_| {
+ internal_datafusion_err!(
+ "Rows from RowCursorStream is still in use by consumer"
+ )
+ }),
+ // Nothing retained yet, or already released, so start over.
+ None => Ok(converter.empty_rows(0, 0)),
+ }
}
- // save the Rows
- fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) {
+
+ /// Account for a freshly built buffer, and retain it for reuse.
+ ///
+ /// The reservation is mandatory rather than best-effort. The buffer is live in the
+ /// cursor whether or not this slot keeps a handle to it, so declining to reserve
+ /// would hide it from the pool instead of avoiding it, and it frees nothing at this
+ /// point either, since the cursor holds the same `Arc`. Retention on top of the
+ /// reservation is free for the same reason. A pool that cannot cover the buffer
+ /// fails the query here, as it did before the buffer was cached at all.
+ fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) -> Result<()> {
self.inner[stream_idx] = Some(Arc::clone(rows));
+ let retained = self.retained_size();
+ debug_assert!(retained >= self.reservation.size());
+ if let Err(e) = self.reservation.try_resize(retained) {
+ self.inner[stream_idx] = None;
+ return Err(e);
+ }
+ Ok(())
+ }
+
+ // drop whatever a finished stream was holding
+ fn release(&mut self, stream_idx: usize) {
+ debug_assert!(self.reservation.size() >= self.retained_size());
+ if let Some(rows) = self.inner[stream_idx].take() {
+ self.reservation.shrink(rows.size());
+ }
+ }
+
+ fn retained_size(&self) -> usize {
Review Comment:
nit: this walks every partition on each batch. That's negligible for typical fan-in, but tracking each slot's size and resizing by the difference would keep `save` O(1) for very wide merges.
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -94,25 +94,62 @@ impl FusedStreams {
}
}
-/// An `Arc<Rows>` that can be reused
+/// An `Arc<Rows>` that can be reused.
+///
+/// Owns the reservation covering every retained buffer for as long as it is
+/// retained, so the cache is visible to the pool rather than held off-book.
+/// A retained buffer keeps its capacity, so the cost is the high-water mark of
+/// each stream, not the size of the batch currently in flight.
#[derive(Debug)]
struct ReusableRows {
inner: Vec<Option<Arc<Rows>>>,
+ reservation: MemoryReservation,
}
impl ReusableRows {
// return a Rows for writing,
// does not clone if the existing rows can be reused
- fn take_next(&mut self, stream_idx: usize) -> Result<Rows> {
- Arc::try_unwrap(self.inner[stream_idx].take().unwrap()).map_err(|_| {
- internal_datafusion_err!(
- "Rows from RowCursorStream is still in use by consumer"
- )
- })
+ fn take_next(&mut self, stream_idx: usize, converter: &RowConverter) -> Result<Rows> {
+ match self.inner[stream_idx].take() {
+ Some(rows) => Arc::try_unwrap(rows).map_err(|_| {
+ internal_datafusion_err!(
+ "Rows from RowCursorStream is still in use by consumer"
+ )
+ }),
+ // Nothing retained yet, or already released, so start over.
+ None => Ok(converter.empty_rows(0, 0)),
+ }
}
- // save the Rows
- fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) {
+
+ /// Account for a freshly built buffer, and retain it for reuse.
+ ///
+ /// The reservation is mandatory rather than best-effort. The buffer is live in the
+ /// cursor whether or not this slot keeps a handle to it, so declining to reserve
+ /// would hide it from the pool instead of avoiding it, and it frees nothing at this
+ /// point either, since the cursor holds the same `Arc`. Retention on top of the
+ /// reservation is free for the same reason. A pool that cannot cover the buffer
+ /// fails the query here, as it did before the buffer was cached at all.
+ fn save(&mut self, stream_idx: usize, rows: &Arc<Rows>) -> Result<()> {
self.inner[stream_idx] = Some(Arc::clone(rows));
+ let retained = self.retained_size();
+ debug_assert!(retained >= self.reservation.size());
+ if let Err(e) = self.reservation.try_resize(retained) {
Review Comment:
If `try_resize` fails here, the slot becomes `None` but the reservation still covers the slot's previous buffer, which is freed when the caller drops `rows`. A failure in `take_next` or `converter.append` leaves the same state. The merge aborts on the error, so today this only over-reports until the stream is dropped. But a later `save` would trip the `debug_assert!` above. Shrinking to `retained_size()` whenever a slot is cleared would keep the reservation and the retained size equal on every path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4266640292
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -587,4 +635,211 @@ mod tests {
Ok(())
}
+
+ /// Drives a `RowCursorStream` over several partitions and reports the pool
+ /// reservation, so the tests below can watch it move.
+ struct RowCursorHarness {
+ stream: RowCursorStream,
+ pool: Arc<dyn MemoryPool>,
+ schema: SchemaRef,
+ rows: usize,
+ str_width: usize,
+ }
+
+ /// Every batch has the same row count and string width, so each converts to
+ /// a `Rows` buffer of the same size regardless of `seq`.
+ fn make_batch(
+ schema: &SchemaRef,
+ seq: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> RecordBatch {
+ let base = (seq * rows) as i32;
+ let a = Int32Array::from_iter_values((0..rows as i32).map(|i| base + i));
+ let b = StringArray::from_iter_values((0..rows).map(|_| "x".repeat(str_width)));
+ RecordBatch::try_new(Arc::clone(schema), vec![Arc::new(a), Arc::new(b)]).unwrap()
+ }
+
+ impl RowCursorHarness {
+ /// `batches_per_partition` batches of `rows` rows each, per partition.
+ /// The sort key spans two columns so this takes the `Rows` path rather
+ /// than the specialized single-column `FieldCursorStream`.
+ fn new(
+ partitions: usize,
+ batches_per_partition: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> Result<Self> {
+ let schema: SchemaRef = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Int32, false),
+ Field::new("b", DataType::Utf8, false),
+ ]));
+
+ let streams: Vec<SendableRecordBatchStream> = (0..partitions)
+ .map(|_| {
+ let batches: Vec<RecordBatch> = (0..batches_per_partition)
+ .map(|seq| make_batch(&schema, seq, rows, str_width))
+ .collect();
+ Box::pin(
+ MemoryStream::try_new(batches, Arc::clone(&schema), None)
+ .unwrap(),
+ ) as SendableRecordBatchStream
+ })
+ .collect();
+
+ let expressions = LexOrdering::new(vec![
+ PhysicalSortExpr::new_default(col("a", &schema)?),
+ PhysicalSortExpr::new_default(col("b", &schema)?),
+ ])
+ .unwrap();
+
+ // Generous limit: this measures what is reserved, not what is refused.
+ let pool: Arc<dyn MemoryPool> =
+ Arc::new(GreedyMemoryPool::new(256 * 1024 * 1024));
+ let reservation = MemoryConsumer::new("test").register(&pool);
+ let stream =
+ RowCursorStream::try_new(&schema, &expressions, streams, reservation)?;
+
+ Ok(Self {
+ stream,
+ pool,
+ schema,
+ rows,
+ str_width,
+ })
+ }
+
+ fn reserved(&self) -> usize {
+ self.pool.reserved()
+ }
+
+ /// Converts one batch with a standalone converter over the same sort
+ /// key, returning `(converter size, size of one retained Rows buffer)`.
+ /// These are what the stream should reserve for its converter and for
+ /// each partition's cached buffer.
+ fn expected_sizes(&self) -> Result<(usize, usize)> {
+ let converter = RowConverter::new(vec![
+ SortField::new(DataType::Int32),
+ SortField::new(DataType::Utf8),
+ ])?;
+ let batch = make_batch(&self.schema, 0, self.rows, self.str_width);
+ let mut rows = converter.empty_rows(0, 0);
+ converter.append(&mut rows, batch.columns())?;
+ Ok((converter.size(), rows.size()))
+ }
+
+ /// Polls `stream_idx`, handing back the cursor so the caller controls when
+ /// it is dropped. `None` once the partition is finished.
+ fn poll_cursor(&mut self, stream_idx: usize) -> Result<Option<RowValues>> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => Ok(Some(values)),
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(None),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+
+ /// Polls `stream_idx` and immediately drops the returned cursor, which is
+ /// what the merge does once a cursor is exhausted. Returns `false` once
+ /// the partition is finished.
+ fn poll_and_drop_cursor(&mut self, stream_idx: usize) -> Result<bool> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => {
+ drop(values);
+ Ok(true)
+ }
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(false),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+ }
+
+ // `RowCursorStream` keeps one `Rows` buffer per partition alive between
+ // polls so the allocation can be reused. That buffer is real memory, and it
+ // outlives the cursor handed to the consumer: the merge drops a cursor as
+ // soon as it is exhausted, but the buffer behind it stays cached.
+ //
+ // So dropping the cursor must not change what the pool reports. If it does,
+ // the bytes still held by the cache have gone off the books and the pool is
+ // under-counting a live allocation.
+ #[test]
+ fn dropping_a_cursor_does_not_unaccount_its_retained_buffer() -> Result<()> {
+ let mut harness = RowCursorHarness::new(1, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ assert_eq!(harness.reserved(), 0);
+
+ let cursor = harness.poll_cursor(0)?.expect("first batch");
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ // The cached `Rows` buffer outlives the cursor, so dropping the cursor
+ // must not release its bytes.
+ drop(cursor);
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ Ok(())
+ }
+
+ // The retained bytes are per partition, so what the pool reports has to
+ // scale with partition count rather than staying flat at the size of
+ // whichever batch happens to be in flight.
+ #[test]
+ fn retained_row_buffer_reservation_scales_with_partitions() -> Result<()> {
+ for partitions in [2, 16] {
+ let mut harness = RowCursorHarness::new(partitions, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ for stream_idx in 0..partitions {
+ assert!(harness.poll_and_drop_cursor(stream_idx)?);
+ }
+ // One shared converter, plus one cached buffer per partition.
+ assert_eq!(
+ harness.reserved(),
+ converter_size + partitions * rows_size,
+ "{partitions} partitions"
+ );
+ }
+
+ Ok(())
+ }
+
+ // The flip side: a partition that will never be polled again must hand its
+ // buffer back. Otherwise the reservation only ever grows and a long merge
+ // holds every partition's high-water mark until the whole stream drops.
+ #[test]
+ fn exhausted_partitions_release_their_retained_buffers() -> Result<()> {
Review Comment:
Could we add a small, large, small batch sequence while another partition keeps its buffer cached? This would verify that the reservation grows with the actual retained `Rows::size()`, stays at the high-water mark after the smaller batch, and shrinks correctly on exhaustion without affecting the other partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Not all rows are accounted in RowCursorStream [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #25372:
URL: https://github.com/apache/datafusion/pull/25372#discussion_r4271146488
##########
datafusion/physical-plan/src/sorts/stream.rs:
##########
@@ -587,4 +635,211 @@ mod tests {
Ok(())
}
+
+ /// Drives a `RowCursorStream` over several partitions and reports the pool
+ /// reservation, so the tests below can watch it move.
+ struct RowCursorHarness {
+ stream: RowCursorStream,
+ pool: Arc<dyn MemoryPool>,
+ schema: SchemaRef,
+ rows: usize,
+ str_width: usize,
+ }
+
+ /// Every batch has the same row count and string width, so each converts to
+ /// a `Rows` buffer of the same size regardless of `seq`.
+ fn make_batch(
+ schema: &SchemaRef,
+ seq: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> RecordBatch {
+ let base = (seq * rows) as i32;
+ let a = Int32Array::from_iter_values((0..rows as i32).map(|i| base + i));
+ let b = StringArray::from_iter_values((0..rows).map(|_| "x".repeat(str_width)));
+ RecordBatch::try_new(Arc::clone(schema), vec![Arc::new(a), Arc::new(b)]).unwrap()
+ }
+
+ impl RowCursorHarness {
+ /// `batches_per_partition` batches of `rows` rows each, per partition.
+ /// The sort key spans two columns so this takes the `Rows` path rather
+ /// than the specialized single-column `FieldCursorStream`.
+ fn new(
+ partitions: usize,
+ batches_per_partition: usize,
+ rows: usize,
+ str_width: usize,
+ ) -> Result<Self> {
+ let schema: SchemaRef = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Int32, false),
+ Field::new("b", DataType::Utf8, false),
+ ]));
+
+ let streams: Vec<SendableRecordBatchStream> = (0..partitions)
+ .map(|_| {
+ let batches: Vec<RecordBatch> = (0..batches_per_partition)
+ .map(|seq| make_batch(&schema, seq, rows, str_width))
+ .collect();
+ Box::pin(
+ MemoryStream::try_new(batches, Arc::clone(&schema), None)
+ .unwrap(),
+ ) as SendableRecordBatchStream
+ })
+ .collect();
+
+ let expressions = LexOrdering::new(vec![
+ PhysicalSortExpr::new_default(col("a", &schema)?),
+ PhysicalSortExpr::new_default(col("b", &schema)?),
+ ])
+ .unwrap();
+
+ // Generous limit: this measures what is reserved, not what is refused.
+ let pool: Arc<dyn MemoryPool> =
+ Arc::new(GreedyMemoryPool::new(256 * 1024 * 1024));
+ let reservation = MemoryConsumer::new("test").register(&pool);
+ let stream =
+ RowCursorStream::try_new(&schema, &expressions, streams, reservation)?;
+
+ Ok(Self {
+ stream,
+ pool,
+ schema,
+ rows,
+ str_width,
+ })
+ }
+
+ fn reserved(&self) -> usize {
+ self.pool.reserved()
+ }
+
+ /// Converts one batch with a standalone converter over the same sort
+ /// key, returning `(converter size, size of one retained Rows buffer)`.
+ /// These are what the stream should reserve for its converter and for
+ /// each partition's cached buffer.
+ fn expected_sizes(&self) -> Result<(usize, usize)> {
+ let converter = RowConverter::new(vec![
+ SortField::new(DataType::Int32),
+ SortField::new(DataType::Utf8),
+ ])?;
+ let batch = make_batch(&self.schema, 0, self.rows, self.str_width);
+ let mut rows = converter.empty_rows(0, 0);
+ converter.append(&mut rows, batch.columns())?;
+ Ok((converter.size(), rows.size()))
+ }
+
+ /// Polls `stream_idx`, handing back the cursor so the caller controls when
+ /// it is dropped. `None` once the partition is finished.
+ fn poll_cursor(&mut self, stream_idx: usize) -> Result<Option<RowValues>> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => Ok(Some(values)),
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(None),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+
+ /// Polls `stream_idx` and immediately drops the returned cursor, which is
+ /// what the merge does once a cursor is exhausted. Returns `false` once
+ /// the partition is finished.
+ fn poll_and_drop_cursor(&mut self, stream_idx: usize) -> Result<bool> {
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ match self.stream.poll_next(&mut cx, stream_idx) {
+ Poll::Ready(Some(Ok((values, _batch)))) => {
+ drop(values);
+ Ok(true)
+ }
+ Poll::Ready(Some(Err(e))) => Err(e),
+ Poll::Ready(None) => Ok(false),
+ Poll::Pending => unreachable!("MemoryStream is never pending"),
+ }
+ }
+ }
+
+ // `RowCursorStream` keeps one `Rows` buffer per partition alive between
+ // polls so the allocation can be reused. That buffer is real memory, and it
+ // outlives the cursor handed to the consumer: the merge drops a cursor as
+ // soon as it is exhausted, but the buffer behind it stays cached.
+ //
+ // So dropping the cursor must not change what the pool reports. If it does,
+ // the bytes still held by the cache have gone off the books and the pool is
+ // under-counting a live allocation.
+ #[test]
+ fn dropping_a_cursor_does_not_unaccount_its_retained_buffer() -> Result<()> {
+ let mut harness = RowCursorHarness::new(1, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ assert_eq!(harness.reserved(), 0);
+
+ let cursor = harness.poll_cursor(0)?.expect("first batch");
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ // The cached `Rows` buffer outlives the cursor, so dropping the cursor
+ // must not release its bytes.
+ drop(cursor);
+ assert_eq!(harness.reserved(), converter_size + rows_size);
+
+ Ok(())
+ }
+
+ // The retained bytes are per partition, so what the pool reports has to
+ // scale with partition count rather than staying flat at the size of
+ // whichever batch happens to be in flight.
+ #[test]
+ fn retained_row_buffer_reservation_scales_with_partitions() -> Result<()> {
+ for partitions in [2, 16] {
+ let mut harness = RowCursorHarness::new(partitions, 4, 512, 64)?;
+ let (converter_size, rows_size) = harness.expected_sizes()?;
+ for stream_idx in 0..partitions {
+ assert!(harness.poll_and_drop_cursor(stream_idx)?);
+ }
+ // One shared converter, plus one cached buffer per partition.
+ assert_eq!(
+ harness.reserved(),
+ converter_size + partitions * rows_size,
+ "{partitions} partitions"
+ );
+ }
+
+ Ok(())
+ }
+
+ // The flip side: a partition that will never be polled again must hand its
+ // buffer back. Otherwise the reservation only ever grows and a long merge
+ // holds every partition's high-water mark until the whole stream drops.
+ #[test]
+ fn exhausted_partitions_release_their_retained_buffers() -> Result<()> {
Review Comment:
Added a bunch more tests, made the harness a little more flexible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org