You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EmilyMatt (via GitHub)" <gi...@apache.org> on 2025/02/24 13:16:23 UTC
[I] Calling try_resize where error is not acceptible [datafusion]
EmilyMatt opened a new issue, #14851:
URL: https://github.com/apache/datafusion/issues/14851
### Describe the bug
In the GroupedHashAggregateStream, update_memory_reservation() has a try_resize() call, an error returned from this function, is usually handled gracefully, as we are usually in the middle of emitting and can just emit again, or are accumulating and can spill(unless not even a single batch fits in memory, in which case this can be a justified panic).
However, when creating a merge stream using the spill data, there is no viable error fallback, this early exits on error if the memory pool denied the reservation.
Using a custom memory pool, it should be possible to implement a "burst" allocation or something similar, and avoid aborting a query, but this requires the aggregate a "declaration of intentions" of sorts, shouldn't resize() be used in such cases, instead of try_resize()?
This is the problematic line:
https://github.com/apache/datafusion/blob/6c5f214643fe81affcad5f3aa9031f02c4390bf0/datafusion/physical-plan/src/aggregates/row_hash.rs#L1050
I believe in line 910 as well,
`match self.update_memory_reservation() {
// Here we can ignore `insufficient_capacity_err` because we will spill later,
// but at least one batch should fit in the memory
Err(DataFusionError::ResourcesExhausted(_))
if self.group_values.len() >= self.batch_size =>
{
Ok(())
}
other => other,
}`
This should check preemptively if we have a possibility of spilling here, and if not, it should be a resize(), not a try_resize() as update_memory_reservation() does.
But this one actually has a possibility of not aborting so seems more agreeable.
It's possible that there are other design ideas taken into consideration here, but this is my belief at least.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14851:
URL: https://github.com/apache/datafusion/issues/14851#issuecomment-3133654528
I agree that the problem described on this ticket should probably be handled with the cascaded merge solution
However, without a reproducer it is not easy to verify. Maybe we can make some sort of reproducer to verify it is fixed in (what will become) DataFusion 50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) [datafusion]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #14851:
URL: https://github.com/apache/datafusion/issues/14851#issuecomment-3130272587
Would this issue be addressed when the multi-level merge pr gets merged? Or do we still need to address it separately? cc. @2010YOUY01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14851:
URL: https://github.com/apache/datafusion/issues/14851#issuecomment-2679009002
Thanks @EmilyMatt -- I agree with the seeming goal of this ticket that when operating in a memory constrained environment it is best to avoid erroring during the spilling process itself
> It's possible that there are other design ideas taken into consideration here, but this is my belief at least.
> Using a custom memory pool, it should be possible to implement a "burst" allocation or something similar, and avoid aborting a query, but this requires the aggregate a "declaration of intentions" of sorts, shouldn't resize() be used in such cases, instead of try_resize()?
I think another classic solution to this is to reduce the memory requirements (though then also decrease the performance) using a mutli-pass merge, as described on https://github.com/apache/datafusion/issues/14692#issue-2855923565
There is some overlap with the discussions that @2010YOUY01 has been having on this ticket (though that one is related to sorting)
- https://github.com/apache/datafusion/issues/14692
So it may be that your idea of improving the reservation calls can work / improve your usecase
However, to solve the problem of "being able to merge in an arbitrary amount of data / spill files given limited memory" will require multi-level merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org