You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jordepic (via GitHub)" <gi...@apache.org> on 2026/06/09 22:29:47 UTC
[I] Accurately reserve memory in the build side of hash joins [datafusion]
jordepic opened a new issue, #22861:
URL: https://github.com/apache/datafusion/issues/22861
### Describe the bug
HashJoinExec's build side reserves get_record_batch_memory_size(&batch) per collected batch. That function deduplicates shared buffers only within one batch, so when the build input emits zero-copy slices of one larger batch — as GroupedHashAggregateStream does when emitting its result in batch_size chunks — every slice is charged the full parent allocation. An aggregate output of S bytes in n slices reserves n × S for S bytes of physical memory; since the build collection cannot spill, this aborts queries that fit in memory with large headroom.
Observed in DataFusion Comet: 26GB reserved for 136MB resident (1.63M-row build side, ~200 slices), failing against a 16GiB pool share. Reporting each slice's sliced size instead would under-count — a single slice keeps the entire parent buffer alive via Arc — so the correct measure for the collection is the union of unique buffers it retains.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Accurately reserve memory in the build side of hash joins [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed issue #22861: Accurately reserve memory in the build side of hash joins
URL: https://github.com/apache/datafusion/issues/22861
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Accurately reserve memory in the build side of hash joins [datafusion]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on issue #22861:
URL: https://github.com/apache/datafusion/issues/22861#issuecomment-4669500036
This turned out to be a duplicate of #22526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org