You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2024/10/15 21:19:25 UTC
[I] Add spilling support for HashJoin [datafusion]
comphead opened a new issue, #12952:
URL: https://github.com/apache/datafusion/issues/12952
### Is your feature request related to a problem or challenge?
It would be helpful to have spilling support for the HashJoin. If there is not enough memory on the machine the join can leverage local disk to spill intermittent results
For reference the spilling support for SortMergeJoin
https://github.com/apache/datafusion/issues/9359
Some ideas was also covered in
https://github.com/apache/datafusion/issues/1599
More reading also https://facebookincubator.github.io/velox/develop/spilling.html
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add spilling support for HashJoin [datafusion]
Posted by "dmitrybugakov (via GitHub)" <gi...@apache.org>.
dmitrybugakov commented on issue #12952:
URL: https://github.com/apache/datafusion/issues/12952#issuecomment-2440125370
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add spilling support for HashJoin [datafusion]
Posted by "demetribu (via GitHub)" <gi...@apache.org>.
demetribu commented on issue #12952:
URL: https://github.com/apache/datafusion/issues/12952#issuecomment-2493981246
test query
```
./datafusion-cli/target/debug/datafusion-cli -m 1m
DataFusion CLI v43.0.0
> set datafusion.optimizer.prefer_hash_join = true;
0 row(s) fetched.
Elapsed 0.003 seconds.
> select * from (select unnest(range(0, 100000)) id) t inner join (select unnest(range(0, 100000)) id) t1 on t.id = t1.id;
External error: Resources exhausted: Failed to allocate additional 278584 bytes for HashJoinInput[13] with 58744 bytes already allocated for this reservation - 123240 bytes remain available for the total pool
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add spilling support for HashJoin [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12952: Add spilling support for HashJoin
URL: https://github.com/apache/datafusion/issues/12952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add spilling support for HashJoin [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #12952:
URL: https://github.com/apache/datafusion/issues/12952#issuecomment-2415189582
@andygrove @viirya cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add spilling support for HashJoin [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12952:
URL: https://github.com/apache/datafusion/issues/12952#issuecomment-5462702803
@jayzhan211 consolidated this project / plan in
- https://github.com/apache/datafusion/issues/24768
Let's focus our discussion there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org