You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "LiaCastaneda (via GitHub)" <gi...@apache.org> on 2025/04/02 14:18:38 UTC
[D] Is HashAggregation spilling enabled by default? [datafusion]
GitHub user LiaCastaneda created a discussion: Is HashAggregation spilling enabled by default?
👋 Hi, I had the doubt if `HashAggregation` has spilling enabled by default (didn't report this as an issue since I saw #7400 ). We are running some heavy queries and are encountering OOM errors:
`Resources exhausted: Failed to allocate additional 10935575608 bytes for GroupedHashAggregateStream[3] with 0 bytes already allocated for this reservation - 9130676356 bytes remain available for the total pool`
I looked at AggregateExec metrics and see no spilling is being made.
GitHub link: https://github.com/apache/datafusion/discussions/15543
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda edited a comment on the discussion: Is HashAggregation spilling enabled by default?
So, I understand is that this is not an OOM error but a memory accounting error? in any case I think there is no fallback spilling being made since I have:
spill_count = 0
spilled_bytes=0
spilled_rows=0
But yeah I will try share a reproducer for this
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12765109
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
GitHub user 2010YOUY01 added a comment to the discussion: Is HashAggregation spilling enabled by default?
Yes, this is an error thrown by DataFusion (instead of OS) during internal memory tracking updates.
The stats will only be updated after the first spill file is written with success, if anything goes wrong during writing the first spill, it's possible to get an error with such 0 stats.
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12772786
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
GitHub user 2010YOUY01 added a comment to the discussion: Is HashAggregation spilling enabled by default?
Spilling is always enabled, the difference is:
- If no memory limit is specified (default), memory accounting inside operators never fails.
- If a memory limit is specified, memory accounting might fail, and a fallback spilling path will be triggered.
It's possible to trigger an error (or bug) before the first spilling file is written, perhaps you can share the reproducer by opening an issue.
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12710267
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda closed a discussion: Is HashAggregation spilling enabled by default?
👋 Hi, I had the doubt if `HashAggregation` has spilling enabled by default (didn't report this as an issue since I saw #7400 ). We are running some heavy queries and are encountering OOM errors:
`Resources exhausted: Failed to allocate additional 10935575608 bytes for GroupedHashAggregateStream[3] with 0 bytes already allocated for this reservation - 9130676356 bytes remain available for the total pool`
I looked at AggregateExec metrics and see no spilling is being made.
GitHub link: https://github.com/apache/datafusion/discussions/15543
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda added a comment to the discussion: Is HashAggregation spilling enabled by default?
Thanks for the answer, yes this makes sense & that's exactly what is happening to me, when I set a concrete memory pool it fails otherwise using a `UnboundedMemoryPool` it works just fine
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12764463
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda added a comment to the discussion: Is HashAggregation spilling enabled by default?
So, I understand is that this is not an OOM error but a memory accounting error?
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12765109
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
GitHub user 2010YOUY01 added a comment to the discussion: Is HashAggregation spilling enabled by default?
Also, just a reminder: external execution is still experimental, so bugs are likely. We plan to improve it over the coming months.
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12772791
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Is HashAggregation spilling enabled by default? [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda added a comment to the discussion: Is HashAggregation spilling enabled by default?
I see, I also forgot to mention Im using a `GreedyMemoryPool` so that might not be the best one to handle queries that need to spillI guess 😄 guiding myself on https://datafusion.apache.org/comet/user-guide/tuning.html (i'm not using cometh though)
GitHub link: https://github.com/apache/datafusion/discussions/15543#discussioncomment-12775706
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org