You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by CrazyJvm <gi...@git.apache.org> on 2014/07/02 09:23:41 UTC
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
GitHub user CrazyJvm opened a pull request:
https://github.com/apache/spark/pull/1282
fix spark.yarn.max.executor.failures explaination
According to
'''scala
private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
''''
default value should be numExecutors * 2, with minimum of 3, and it's same to the config
`spark.yarn.max.worker.failures`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CrazyJvm/spark yarn-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1282.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1282
----
commit a4b2e27b0c2d2345a60ba66943b219968465b48a
Author: CrazyJvm <cr...@gmail.com>
Date: 2014-07-02T06:59:48Z
fix configuration spark.yarn.max.executor.failures
commit 2900d234c6ebb90a5c4601083ddf8d329a2ee99d
Author: CrazyJvm <cr...@gmail.com>
Date: 2014-07-02T07:04:51Z
fix style
commit 211f1302aa6d57b07a7b2d3b7cd4ab21e6d50bbd
Author: CrazyJvm <cr...@gmail.com>
Date: 2014-07-02T07:06:28Z
fix html tag
commit 86effa612d2ec9ae991b43e229c4ed266e6605a6
Author: CrazyJvm <cr...@gmail.com>
Date: 2014-07-02T07:15:08Z
change expression
commit c438aecdec8ce90cb839b7c9aa8260ff4d3c62ba
Author: CrazyJvm <cr...@gmail.com>
Date: 2014-07-02T07:18:18Z
fix style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1282#discussion_r14614205
--- Diff: docs/running-on-yarn.md ---
@@ -55,9 +55,10 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.max.executor.failures</code></td>
- <td>2*numExecutors</td>
+ <td>numExecutors * 2, with minimum of 3</td>
<td>
- The maximum number of executor failures before failing the application.
+ It's the same to <code>spark.yarn.max.worker.failures</code>,
--- End diff --
we haven't been listing deprecated configs as we don't want people to continue to use them.
It would be nice if we had somewhere people could go to see the list of deprecated configs and new mappings though. But that is a separate jira.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48274131
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-47747966
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-47747968
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16292/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2400 : fix spark.yarn.max.executor.failu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48276380
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2400 : fix spark.yarn.max.executor.failu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48276382
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16395/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-47744840
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2400 : fix spark.yarn.max.executor.failu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1282
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-47744848
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2400 : fix spark.yarn.max.executor.failu...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48384242
Thanks looks good. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48221848
Please file a jira to match this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1282#issuecomment-48274137
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---