You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2025/01/30 10:07:55 UTC
[I] using off heap with spilling [datafusion-comet]
rluvaton opened a new issue, #1353:
URL: https://github.com/apache/datafusion-comet/issues/1353
In the tuning guide I see the recommended way is to use off heap:
https://github.com/apache/datafusion-comet/blob/e9649477c4f8b4c6906244c3cc6828b83f32f735/docs/source/user-guide/tuning.md?plain=1#L28-L29
But I also see that it does not support spilling (which is not written in the tuning guide), if so, how is it the recommended way?
https://github.com/apache/datafusion-comet/blob/e9649477c4f8b4c6906244c3cc6828b83f32f735/spark/src/main/java/org/apache/spark/CometTaskMemoryManager.java#L56-L60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] using off heap with spilling [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1353: using off heap with spilling
URL: https://github.com/apache/datafusion-comet/issues/1353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] using off heap with spilling [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1353:
URL: https://github.com/apache/datafusion-comet/issues/1353#issuecomment-2625484525
Spilling is supported in the native plan (to the extent that DataFusion currently supports spilling). Spark can trigger spilling in the native plan, but the native plan cannot trigger spilling in Spark JVM operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org