You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by devaraj-kavali <gi...@git.apache.org> on 2018/04/17 18:43:03 UTC
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
GitHub user devaraj-kavali opened a pull request:
https://github.com/apache/spark/pull/21088
[SPARK-24003][CORE] Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's
## What changes were proposed in this pull request?
Added support to specify the 'spark.executor.extraJavaOptions' value in terms of the `{{APP_ID}}` and/or `{{EXECUTOR_ID}}`, `{{APP_ID}}` will be replaced by Application Id and `{{EXECUTOR_ID}}` will be replaced by Executor Id while starting the executor.
## How was this patch tested?
I have verified this by checking the executor process command and gc logs. I verified the same in different deployment modes(Standalone, YARN, Mesos) client and cluster modes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devaraj-kavali/spark SPARK-24003
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21088.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21088
----
commit ff6e79315c0ea26b45207753d3e79f96f2395329
Author: Devaraj K <de...@...>
Date: 2018-04-17T18:37:11Z
[SPARK-24003][CORE] Add support to provide spark.executor.extraJavaOptions
in terms of App Id and/or Executor Id's
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali commented on the issue:
https://github.com/apache/spark/pull/21088
Thanks @jerryshao and @jiangxb1987 for looking into this.
I have updated with k8s support and fixed review comment, please have a look into this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89904/testReport)** for PR 21088 at commit [`1758577`](https://github.com/apache/spark/commit/17585773104fa0169f6e162b656873a3ed11786a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89835/testReport)** for PR 21088 at commit [`5dd2b87`](https://github.com/apache/spark/commit/5dd2b87e0745f5b2c2d2b014859a818725d700d2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21088
ok to test.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r182765201
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala ---
@@ -142,7 +142,14 @@ private[deploy] class ExecutorRunner(
private def fetchAndRunExecutor() {
try {
// Launch the process
- val builder = CommandUtils.buildProcessBuilder(appDesc.command, new SecurityManager(conf),
+ val subsCommand = Command(appDesc.command.mainClass,
--- End diff --
How about
```
val subsOpts = appDesc.command.javaOpts.map {
opt => Utils.substituteAppNExecIds(opt, appId, execId.toString)
}
val subsCommand = appDesc.command.copy(javaOpts = subsOpts)
``` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89819/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21088
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r184542781
--- Diff: docs/configuration.md ---
@@ -328,6 +328,11 @@ Apart from these, the following properties are also available, and may be useful
Note that it is illegal to set Spark properties or maximum heap size (-Xmx) settings with this
option. Spark properties should be set using a SparkConf object or the spark-defaults.conf file
used with the spark-submit script. Maximum heap size settings can be set with spark.executor.memory.
+
+ The following symbols, if present will be interpolated: {{APP_ID}} will be replaced by
+ Application Id and {{EXECUTOR_ID}} will be replaced by Executor Id. For example, to enable
+ verbose gc logging to a file named for the Executor Id of the App in /tmp, pass a 'value' of:
--- End diff --
nit: "executor ID", "app"
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89908/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89904/testReport)** for PR 21088 at commit [`1758577`](https://github.com/apache/spark/commit/17585773104fa0169f6e162b656873a3ed11786a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89945/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89945/testReport)** for PR 21088 at commit [`a7f35f4`](https://github.com/apache/spark/commit/a7f35f4c782d76b78d26688ec9a593d2bbbf3c39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89819/testReport)** for PR 21088 at commit [`5dd2b87`](https://github.com/apache/spark/commit/5dd2b87e0745f5b2c2d2b014859a818725d700d2).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89908/testReport)** for PR 21088 at commit [`27cc29c`](https://github.com/apache/spark/commit/27cc29cd62b704318ceddde82e279f7bd7f6e15f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89909/testReport)** for PR 21088 at commit [`932b7d1`](https://github.com/apache/spark/commit/932b7d197d12f0fe714f369d27482d402ac4a695).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by devaraj-kavali <gi...@git.apache.org>.
Github user devaraj-kavali commented on the issue:
https://github.com/apache/spark/pull/21088
For standalone cluster(`DriverRunner.scala:182`), there is driverId which we can use here and for Mesos cluster(`MesosRestServer.scala:107`), there is submission ID available but there is no app ID available in these two cases. In k8s mode, it requires some change to do this(by updating the existing conf with substituted val) not sure whether it is worth changing this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89835/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89835/testReport)** for PR 21088 at commit [`5dd2b87`](https://github.com/apache/spark/commit/5dd2b87e0745f5b2c2d2b014859a818725d700d2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r184814790
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -914,7 +916,9 @@ private[spark] class Client(
s"(was '$opts'). Use spark.yarn.am.memory instead."
throw new SparkException(msg)
}
- javaOpts ++= Utils.splitCommandString(opts).map(YarnSparkHadoopUtil.escapeForShell)
+ javaOpts ++= Utils.splitCommandString(opts)
+ .map(Utils.substituteAppId(_, appId.toString))
--- End diff --
nit: indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21088
Should you also support this in kubernetes deploy mode?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21088
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21088
Assuming the {{APP_ID}}/{{EXECUTOR_ID}} would be useful under some meaningful scenarios, the changes looks fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89908/testReport)** for PR 21088 at commit [`27cc29c`](https://github.com/apache/spark/commit/27cc29cd62b704318ceddde82e279f7bd7f6e15f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r184543396
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -892,7 +892,8 @@ private[spark] class Client(
// Include driver-specific java options if we are launching a driver
if (isClusterMode) {
sparkConf.get(DRIVER_JAVA_OPTIONS).foreach { opts =>
- javaOpts ++= Utils.splitCommandString(opts).map(YarnSparkHadoopUtil.escapeForShell)
+ javaOpts ++= Utils.splitCommandString(Utils.substituteAppId(opts, appId.toString))
--- End diff --
I'd prefer:
```
Utils.splitCommandString(opts)
.map(Utils.substituteAppId(...))
.map(YarnSparkHadoopUtil.escapeForShell)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r184543613
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala ---
@@ -142,7 +142,11 @@ private[deploy] class ExecutorRunner(
private def fetchAndRunExecutor() {
try {
// Launch the process
- val builder = CommandUtils.buildProcessBuilder(appDesc.command, new SecurityManager(conf),
+ val subsOpts = appDesc.command.javaOpts.map {
+ opt => Utils.substituteAppNExecIds(opt, appId, execId.toString)
--- End diff --
nit: `opt =>` goes in previous line
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89819/testReport)** for PR 21088 at commit [`5dd2b87`](https://github.com/apache/spark/commit/5dd2b87e0745f5b2c2d2b014859a818725d700d2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89909/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21088
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21088
It would be nice to add this to YARN's AM too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89904/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21088
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21088
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89909/testReport)** for PR 21088 at commit [`932b7d1`](https://github.com/apache/spark/commit/932b7d197d12f0fe714f369d27482d402ac4a695).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21088#discussion_r184542727
--- Diff: docs/configuration.md ---
@@ -328,6 +328,11 @@ Apart from these, the following properties are also available, and may be useful
Note that it is illegal to set Spark properties or maximum heap size (-Xmx) settings with this
option. Spark properties should be set using a SparkConf object or the spark-defaults.conf file
used with the spark-submit script. Maximum heap size settings can be set with spark.executor.memory.
+
+ The following symbols, if present will be interpolated: {{APP_ID}} will be replaced by
+ Application Id and {{EXECUTOR_ID}} will be replaced by Executor Id. For example, to enable
--- End diff --
nit: "application ID", "executor ID".
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21088
**[Test build #89945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89945/testReport)** for PR 21088 at commit [`a7f35f4`](https://github.com/apache/spark/commit/a7f35f4c782d76b78d26688ec9a593d2bbbf3c39).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org