You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pmackles <gi...@git.apache.org> on 2017/10/17 12:02:25 UTC
[GitHub] spark pull request #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not hono...
GitHub user pmackles opened a pull request:
https://github.com/apache/spark/pull/19515
[SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by MesosClusterD…
…ispatcher
## What changes were proposed in this pull request?
Allow JVM max heap size to be controlled for MesosClusterDispatcher via SPARK_DAEMON_MEMORY environment variable.
## How was this patch tested?
Tested on local Mesos cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pmackles/spark SPARK-22287
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19515.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19515
----
commit 33a8e6880a468335330a7cb6507493de8b125faa
Author: Paul Mackles <pm...@adobe.com>
Date: 2017-10-17T11:49:07Z
[SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by MesosClusterDispatcher
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83648/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19515
**[Test build #83648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83648/testReport)** for PR 19515 at commit [`33a8e68`](https://github.com/apache/spark/commit/33a8e6880a468335330a7cb6507493de8b125faa).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by ArtRand <gi...@git.apache.org>.
Github user ArtRand commented on the issue:
https://github.com/apache/spark/pull/19515
Hello @pmackles, thanks for this. It would be set to the value of `SPARK_DRIVER_MEMORY` by default correct? What to you think about introducing a new envvar (`SPARK_DISPATCHER_MEMORY`) so that it can be individually configured? Only motivated by the fact that the Dispatcher is a pretty unique beast compared to the other Spark daemons.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by pmackles <gi...@git.apache.org>.
Github user pmackles commented on the issue:
https://github.com/apache/spark/pull/19515
Hi @ArtRand - Based on my testing and interpretation of the code, ```SPARK_DRIVER_MEMORY``` has no affect on MesosClusterDispatcher. Heap size always winds up being set to ```-Xmx1G``` (the default).
I went with ```SPARK_DAEMON_MEMORY``` because that's what is used for the Master/Worker in a standalone cluster and I think Dispatcher is pretty similar in nature to those daemons. That said, I don't have a strong opinion and ```SPARK_DISPATCHER_MEMORY``` works fine for my use case so I am happy to switch it in the name of getting this PR accepted.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by pmackles <gi...@git.apache.org>.
Github user pmackles commented on the issue:
https://github.com/apache/spark/pull/19515
@felixcheung - any chance of getting this tiny change merged and included in the upcoming 2.2.1 release?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by pmackles <gi...@git.apache.org>.
Github user pmackles commented on the issue:
https://github.com/apache/spark/pull/19515
@ArtRand - WDYT? I was going to switch it to ```SPARK_DISPATCHER_MEMORY``` but then I noticed that the other env vars for MesosClusterDispatcher or also prefixed with ```SPARK_DAEMON_*``` so I thought it might be better to keep the names consistent.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19515
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by pmackles <gi...@git.apache.org>.
Github user pmackles commented on the issue:
https://github.com/apache/spark/pull/19515
@ArtRand - yeah default would remain the same which is 1g
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19515
**[Test build #83648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83648/testReport)** for PR 19515 at commit [`33a8e68`](https://github.com/apache/spark/commit/33a8e6880a468335330a7cb6507493de8b125faa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by ArtRand <gi...@git.apache.org>.
Github user ArtRand commented on the issue:
https://github.com/apache/spark/pull/19515
Yeah that sounds good. Thanks for this!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19515
@pmackles perhaps you could email this to dev@spark.apache.org to get some visibility to this and hopefully someone else on the mesos side can review?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by ArtRand <gi...@git.apache.org>.
Github user ArtRand commented on the issue:
https://github.com/apache/spark/pull/19515
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19515
LGTM, merging to master / 2.2.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19515
Jenkins, test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by ArtRand <gi...@git.apache.org>.
Github user ArtRand commented on the issue:
https://github.com/apache/spark/pull/19515
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19515
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19515
@susanxhuynh or anyone from the mesos side would you please review?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not hono...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19515
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org