You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/03/18 12:08:37 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #5636: Implement memory management / limiting in `SymmetricHashJoinExec`
alamb opened a new issue, #5636:
URL: https://github.com/apache/arrow-datafusion/issues/5636
Actually there is also `SymmetricHashJoinExec` -- but, I suppose, its memory management is a bit more complicated and may be behind the scope of this issue:
- It's, sort of, isolated -- at this moment there are no options for planning this operator with DataFusion planner
- I don't think that simply throwing error and aborting execution is acceptable for `SymmetricHashJoinExec` -- if my understanding is correct -- main use case for this operator is joining two unbounded sources (streaming jobs), and from this point of view it doesn't make much sense to limit memory without any spilling fallbacks (subjectively, it doesn't seem correct to fail data streaming job in case of memory overallocation attempt)
My proposal here would be to file separate issue for `SymmetricHashJoinExec` memory management, and (as I see it) implement memory limitation along with data spilling. Maybe, we can go for it when we have reliable spilling for `HashJoinExec`, however, prior to it is also an option.
@ozankabak, @metesynnada it would be great to hear your thoughts on it.
_Originally posted by @korowa in https://github.com/apache/arrow-datafusion/issues/5220#issuecomment-1474225900_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] mustafasrepo closed issue #5636: Implement memory management / limiting in `SymmetricHashJoinExec`
Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo closed issue #5636: Implement memory management / limiting in `SymmetricHashJoinExec`
URL: https://github.com/apache/arrow-datafusion/issues/5636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] metesynnada commented on issue #5636: Implement memory management / limiting in `SymmetricHashJoinExec`
Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on issue #5636:
URL: https://github.com/apache/arrow-datafusion/issues/5636#issuecomment-1479026628
I will be looking into this in a week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org