You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2026/07/17 08:12:10 UTC
[I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
2010YOUY01 opened a new issue, #23658:
URL: https://github.com/apache/datafusion/issues/23658
### Is your feature request related to a problem or challenge?
Follow up to https://github.com/apache/datafusion/issues/12114
When implementing https://github.com/apache/datafusion/pull/23657, I have found we could add more test coverages for larger-than-memory execution for ordered aggregation, by extending the aggregation fuzzer.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
Posted by "naman-modi (via GitHub)" <gi...@apache.org>.
naman-modi commented on issue #23658:
URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5131140083
Interesting read @nathanb9. The core loop i am thinking of, is close in spirit: random data + plans under a tight budget, spill on/off, and asserting the query either succeeds or fails with an expected OOM/abort (no hang). Also, our checks are more stringent than theirs.
A few things though; it's multi-threaded with a shared budget, while my initial PR (#23979) is single-query per-pool. So running several plans concurrently under one shared FairSpillPool would be a good follow up work. thanks for sharing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #23658:
URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5015041411
The memory constrained fuzz tests I created for sort and aggregate might be a good place to add them if you mean manually written fuzz tests, or to get some ideas from there:
https://github.com/apache/datafusion/blob/67947b6b90c891843eb167da2fd3e562587834c1/datafusion/core/tests/fuzz_cases/spilling_fuzz_in_memory_constrained_env.rs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on issue #23658:
URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5124221933
Might be interesting to check out [Velox's memory arbitration fuzzer](https://github.com/facebookincubator/velox/blob/main/velox/docs/develop/testing/memory-arbitration-fuzzer.rst) for ideas. FYI, it targets interquery arbitration so it's not a exact match
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
Posted by "naman-modi (via GitHub)" <gi...@apache.org>.
naman-modi commented on issue #23658:
URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5116028836
@2010YOUY01, I am planning to add a bounded memory pool knob to the fuzzer's `SessionContextGenerator`, so some randomized contexts run under a bounded `FairSpillPool` and hit the spill paths. Baseline stays unbounded for correct assertions.
selecting the right "memory sizing" to bound to, is the tricky part; I first tried a fraction of the dataset size, but it did not work as aggregate memory tracks distinct groups, not rows, so any pool that does not instantly exhaust sits well above the real footprint and never spills.
So I am experimenting with sizing it off the measured peak instead, which I will get when I run the baseline under a tracking pool, and then force the spill, on the basis of a fraction of this "peak memory". Let me know if you have any opinions/suggestion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Extend aggregation fuzz testing with memory-limit tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #23658:
URL: https://github.com/apache/datafusion/issues/23658#issuecomment-5040969180
Yes, this would be great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org