You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2025/10/06 22:36:36 UTC

Re: [I] Stage level resource management to handle offheap/onheap memory conflict [incubator-gluten]

FelixYBW commented on issue #4392:
URL: https://github.com/apache/incubator-gluten/issues/4392#issuecomment-3374495583

   Maybe it's known feature. We encountered a case in Spark3.5. Once you enable offheap in Spark, Spark will try to use offheap to cache the internalrows, if offheap memory isn't enough, it won't check the onheap memory but just spill the data. So if you config a large on heap memory but small offheap memory, spill will be triggered more frequently.
   
   While you either can't config a large offheap memory and a small onheap memory because not all operators are able to cache the internal rows into offheap. Onheap demand is still very large for stages fallback to vanilla spark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org