You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stefano Parmesan (JIRA)" <ji...@apache.org> on 2015/06/30 00:54:05 UTC
[jira] [Commented] (SPARK-701) Wrong SPARK_MEM setting with
different EC2 master and worker machine types
[ https://issues.apache.org/jira/browse/SPARK-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606596#comment-14606596 ]
Stefano Parmesan commented on SPARK-701:
----------------------------------------
[~shivaram] it looks like this issue reappeared, in a way (talking about 1.4.0 now, not tested on previous versions): if you create a cluster with an {{m1.small}} master (1.7GB RAM) and one {{m1.large}} worker (7.5GB RAM), {{spark.executor.memory}} will be set to 512MB, and that's because of [system_ram_kb = min(slave_ram_kb, master_ram_kb)|https://github.com/mesos/spark-ec2/blob/e642aa362338e01efed62948ec0f063d5fce3242/deploy_templates.py#L32]
Quite often you use a smaller master instance compared to workers; smaller means fewer RAM, and the line of code I linked above shows that the minimum between the master and the worker(s) memory is used as {{spark_mb}}, which in turn is used as {{default_spark_mem}} to generate the [spark-defaults.conf|https://github.com/mesos/spark-ec2/blob/e642aa362338e01efed62948ec0f063d5fce3242/templates/root/spark/conf/spark-defaults.conf].
If we read the title of this bug, it would seem like the issue reappeared, and we should reopen it; if we instead read the description, we'll notice it's not 100% the same issue: it says "SPARK_MEM in spark-env.sh is set based on the master's memory instead of the worker's", while now "it's set on the minimum between the master's memory and the worker's, which is quite often the master's".
What do you suggest? Should we reopen this three-digits issue, or create a new one?
> Wrong SPARK_MEM setting with different EC2 master and worker machine types
> --------------------------------------------------------------------------
>
> Key: SPARK-701
> URL: https://issues.apache.org/jira/browse/SPARK-701
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Affects Versions: 0.7.0
> Reporter: Josh Rosen
> Assignee: Shivaram Venkataraman
> Fix For: 0.7.0
>
>
> When launching a spark-ec2 cluster using different worker and master machine types, SPARK_MEM in spark-env.sh is set based on the master's memory instead of the worker's. This causes jobs to hang if the master has more memory than the workers (because jobs will request too much memory).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org