You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2015/02/06 00:53:03 UTC
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/4409
[SPARK-5636] Ramp up faster in dynamic allocation
A recent patch #4051 made the initial number default to 0. Any Spark application using dynamic allocation's default settings will ramp up very slowly. Since we never request more executors than needed to saturate the pending tasks, it is safe to ramp up quickly. The current default of 60 may be too slow.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark dynamic-allocation-interval
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4409.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4409
----
commit d3cc485a62be3959171fd876856b5807f68ea1db
Author: Andrew Or <an...@databricks.com>
Date: 2015-02-05T23:50:01Z
Lower request interval
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73163771
[Test build #26867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26867/consoleFull) for PR 4409 at commit [`d3cc485`](https://github.com/apache/spark/commit/d3cc485a62be3959171fd876856b5807f68ea1db).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `protected class CaseInsensitiveMap(map: Map[String, String]) extends Map[String, String]`
* `trait CreatableRelationProvider `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73156052
[Test build #26867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26867/consoleFull) for PR 4409 at commit [`d3cc485`](https://github.com/apache/spark/commit/d3cc485a62be3959171fd876856b5807f68ea1db).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4409#discussion_r24262314
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -78,7 +78,7 @@ private[spark] class ExecutorAllocationManager(
// How long there must be backlogged tasks for before an addition is triggered
--- End diff --
k I fixed this when I merged it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by ksakellis <gi...@git.apache.org>.
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/4409#discussion_r24224468
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -78,7 +78,7 @@ private[spark] class ExecutorAllocationManager(
// How long there must be backlogged tasks for before an addition is triggered
--- End diff --
nit: Maybe mention in the comment that this is measured in seconds - not clear from the config name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73155467
@sryza
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73155754
I think this is a good idea. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73188040
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by ksakellis <gi...@git.apache.org>.
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73187911
one small comment thing but other than that LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4409#issuecomment-73163777
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26867/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5636] Ramp up faster in dynamic allocat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4409
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org