You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "xinghuayu007 (via GitHub)" <gi...@apache.org> on 2025/08/08 09:25:13 UTC
[I] Adjust heap memory dynamically maybe cause high memory consumption [incubator-gluten]
xinghuayu007 opened a new issue, #10388:
URL: https://github.com/apache/incubator-gluten/issues/10388
### Description
GLUTEN-8018 increases heap memory when the fallback ratio over spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold may cause high memory consumption. For example, 10G for heap memory, and 15G for off-heap memory, using the follow setting.
'
spark.gluten.auto.adjustStageResource.enabled=true;
spark.gluten.auto.adjustStageResources.heap.ratio=3.0d;
spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold=0.1d;
'
When this rule is applied, heap memory will become 30G, and the total memory for gluten is 45G, that brings 20G consumption and causes OOM.
In my ideal, it's better to decrease off-heap memory at the same time when increasing heap memory to make the total memory consumption no-increasing.
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Adjust heap memory dynamically maybe cause high memory consumption [incubator-gluten]
Posted by "ayushi-agarwal (via GitHub)" <gi...@apache.org>.
ayushi-agarwal commented on issue #10388:
URL: https://github.com/apache/incubator-gluten/issues/10388#issuecomment-3183309168
@PHILO-HE One thing which I noticed for this feature is that if for some reason executors are already maxed out and they cannot be decomissioned due to reasons like they have rdds cached on them, and some stage wantes to use a different resource profile then query just hangs there, so is cluster autoscaling also mandatory for this feature to work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Adjust heap memory dynamically maybe cause high memory consumption [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE closed issue #10388: Adjust heap memory dynamically maybe cause high memory consumption
URL: https://github.com/apache/incubator-gluten/issues/10388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Adjust heap memory dynamically maybe cause high memory consumption [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10388:
URL: https://github.com/apache/incubator-gluten/issues/10388#issuecomment-3170376022
Does your usecase use whole stage fallback or per operator fallback? Ideally we should use whole stage fallback to either assign whole memory to offheap or onheap.
Per operator fallback will need double memory in many usecases like a offloaded agg + fallback sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org