You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/01/23 11:54:35 UTC
[D] Maintanence status of the dynamic off-heap sizing feature? [incubator-gluten]
GitHub user zhztheplayer created a discussion: Maintanence status of the dynamic off-heap sizing feature?
Previous PRs about dynamic off-heap sizing:
1. https://github.com/apache/incubator-gluten/pull/5439
2. https://github.com/apache/incubator-gluten/pull/7803
The initial PR 1 was merged in April, and it seems we are still having the [OOM risk](https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2224229862) not solving yet. @supermem613 Do you have any thoughts or updates? Thanks.
@yikf If you also have some use case you are able to share?
cc @zhli1142015 @Yohahaha
GitHub link: https://github.com/apache/incubator-gluten/discussions/8600
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] [VL] Maintanence status of the dynamic off-heap sizing feature? [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
GitHub user zhztheplayer edited a discussion: [VL] Maintanence status of the dynamic off-heap sizing feature?
Previous PRs about dynamic off-heap sizing:
1. https://github.com/apache/incubator-gluten/pull/5439
2. https://github.com/apache/incubator-gluten/pull/7803
The initial PR 1 was merged in April, and it seems we are still having the [OOM risk](https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2224229862) not solving yet. @supermem613 Do you have any thoughts or updates? Thanks.
@yikf If you also have some use case you are able to share?
cc @zhli1142015 @Yohahaha
GitHub link: https://github.com/apache/incubator-gluten/discussions/8600
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [D] [VL] Maintanence status of the dynamic off-heap sizing feature? [incubator-gluten]
Posted by "yikf (via GitHub)" <gi...@apache.org>.
GitHub user yikf added a comment to the discussion: [VL] Maintanence status of the dynamic off-heap sizing feature?
@zhztheplayer Thanks for your invitation.
It has been found that when dynamic memory is enabled, the memory requested by Spark from the resource manager is larger than the available memory. In short, it has restricted itself, and the total amount of requested memory has increased. https://github.com/apache/incubator-gluten/pull/5439#discussion_r1847590308.
GitHub link: https://github.com/apache/incubator-gluten/discussions/8600#discussioncomment-11937579
----
This is an automatically sent email for commits@gluten.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@gluten.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org