You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2015/12/17 08:11:46 UTC
[jira] [Commented] (SPARK-12384) Allow -Xms to be set differently
then -Xmx
[ https://issues.apache.org/jira/browse/SPARK-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061639#comment-15061639 ]
Saisai Shao commented on SPARK-12384:
-------------------------------------
IIUC, there's also another limitation in container level. For example, if we set -Xms to 1g and -Xmx to 10g, also container memory resource is 10g, even if this user only uses 2g, but still other users cannot use the left 8g memory.
So basically current container resource is a fixed number, it cannot be adjusted in the runtime, we have to choose a large value to prevent OOM, and this will actually bring in memory waste. Probably this problem can be handled by container resizing.
> Allow -Xms to be set differently then -Xmx
> ------------------------------------------
>
> Key: SPARK-12384
> URL: https://issues.apache.org/jira/browse/SPARK-12384
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit, YARN
> Affects Versions: 1.6.0
> Reporter: Thomas Graves
>
> Currently Spark automatically sets the -Xms parameter to be the same as the -Xmx parameter. We should allow the user to set this separately.
> The main use case here is if I'm running the spark-shell on a shared gateway. Many users specify a larger memory size then needed and will never use that much memory, so all its doing is preventing other users from potentially using that memory. Allowing it to be less is just more multi-tenant friendly.
> I think it makes sense to leave this for cluster mode, although if a user really wants to override I don't see why we shouldn't let them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org