You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xxhZs (via GitHub)" <gi...@apache.org> on 2025/12/01 09:37:14 UTC
[I] Spill error (oom) with sort merge join [datafusion]
xxhZs opened a new issue, #19013:
URL: https://github.com/apache/datafusion/issues/19013
### Describe the bug
I run some data(300g) in small mem(64g) with sort merge join spill.
Then i get error
`Failed to allocate additional {} for {} with {} already allocated for this reservation - {} remain available for the total pool`
And i try to fix it. I fix two code
remove this code:
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/stream.rs#L176
and
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/sort.rs#L785
to
> loop {
> match self.reservation.try_grow(size) {
> Ok(_) => Ok(()),
> Err(e) => {
> if self.in_mem_batches.is_empty() {
> return Err(Self::err_with_oom_context(e));
> }
>
> // Spill and try again.
> self.sort_and_spill_in_mem_batches().await?;
> }
> }
I think the issues are as follows:
1. The memory was immediately occupied as soon as the sort operator spilled data to disk.
2. The row operator failed to spill data to disk, which caused this problem.
So could we fix it in this way, or are there any better solutions? I'm willing to take on the fix.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spill error (oom) with sort merge join [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #19013:
URL: https://github.com/apache/datafusion/issues/19013#issuecomment-3595537899
Would it be easy to have a reproduced case? Which is helpful to add as a test in your fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spill error (oom) with sort merge join [datafusion]
Posted by "xxhZs (via GitHub)" <gi...@apache.org>.
xxhZs commented on issue #19013:
URL: https://github.com/apache/datafusion/issues/19013#issuecomment-3600360083
> Would it be easy to have a reproduced case? Which is helpful to add as a test in your fix
could you tell me how to write some test data in DataFusion? I'd like to insert some data to run tests on my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org