You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by YanTangZhai <gi...@git.apache.org> on 2014/12/26 12:36:05 UTC
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/3810
[SPARK-4962] [CORE] Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period
When SparkContext object is instantiated, TaskScheduler is started and some resources are allocated from cluster. However, these
resources may be not used for the moment. For example, DAGScheduler.JobSubmitted is processing and so on. These resources are wasted in
this period. Thus, we want to put TaskScheduler.start back to shorten cluster resources occupation period specially for busy cluster.
TaskScheduler could be started just before running stages.
We could analyse and compare the resources occupation period before and after optimization.
TaskScheduler.start execution time: [time1__]
DAGScheduler.JobSubmitted (excluding HadoopRDD.getPartitions or TaskScheduler.start) execution time: [time2_]
HadoopRDD.getPartitions execution time: [time3___]
Stages execution time: [time4_____]
The cluster resources occupation period before optimization is [time2_][time3___][time4_____].
The cluster resources occupation period after optimization is....[time3___][time4_____].
In summary, the cluster resources
occupation period after optimization is less than before.
If HadoopRDD.getPartitions could be put forward (SPARK-4961), the period may be shorten more which is [time4_____].
The resources saving is important for busy cluster.
The main purpose of this PR is to decrease resources waste for busy cluster.
For example, a process initializes a SparkContext instance, reads a few files from HDFS or many records from PostgreSQL, and then calls RDD's collect operation to submit a job.
When SparkContext is initialized, an app is submitted to cluster and some resources are hold by this app.
These resources are not used really until the job is submitted by RDD's action.
The resources in the period from initialization to actual use could be considered wasteful.
If app is submitted when SparkContext is initialized, all of resources needed by the app may be granted before running job.
Then the job could runs efficiently without resource constraint.
On the contrary, if app is submitted when job is submitted, resources needed by the app may be granted at different times. Then the job may run not so efficiently since some resources are applying.
Thus I use a configuration parameter spark.scheduler.app.slowstart (default false) to let user make tradeoffs between economy and efficiency.
There are 9 kinds of master URL and 6 kinds of SchedulerBackend.
LocalBackend and SimrSchedulerBackend don't need to put starting back since there is no difference.
SparkClusterSchedulerBackend (yarn-standalone or yarn-cluster) does not put starting back since the app should be submitted in advance by SparkSubmit.
CoarseMesosSchedulerBackend and MesosSchedulerBackend could put starting back.
YarnClientSchedulerBackend (yarn-client) could put starting back.
This PR puts TaskScheduler.start back only for yarn-client mode in the early.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YanTangZhai/spark SPARK-4962
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3810.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3810
----
commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-06T13:07:08Z
Merge pull request #1 from apache/master
update
commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-20T13:14:08Z
Merge pull request #3 from apache/master
Update
commit 8a0010691b669495b4c327cf83124cabb7da1405
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-12T06:54:58Z
Merge pull request #6 from apache/master
Update
commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-16T12:03:22Z
Merge pull request #7 from apache/master
Update
commit 76d40277d51f709247df1d3734093bf2c047737d
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-10-20T12:52:22Z
Merge pull request #8 from apache/master
update
commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-11-04T09:00:31Z
Merge pull request #9 from apache/master
Update
commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-11-11T03:18:24Z
Merge pull request #10 from apache/master
Update
commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-12-01T11:23:56Z
Merge pull request #11 from apache/master
Update
commit 718afebe364bd54ac33be425e24183eb1c76b5d3
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-12-05T11:08:31Z
Merge pull request #12 from apache/master
update
commit e4c2c0a18bdc78cc17823cbc2adf3926944e6bc5
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-12-24T03:15:22Z
Merge pull request #15 from apache/master
update
commit 05469de9f0482bce54a60161b9cb386a64173826
Author: yantangzhai <ty...@163.com>
Date: 2014-12-26T07:11:30Z
[SPARK-4962] [CORE] Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69698965
[Test build #25458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25458/consoleFull) for PR 3810 at commit [`74094af`](https://github.com/apache/spark/commit/74094afa1bafbc55540f334c3d37fcb3a2570937).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69686970
[Test build #25448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25448/consoleFull) for PR 3810 at commit [`a0153e4`](https://github.com/apache/spark/commit/a0153e4af52328296b85fa2d1bd2bd69ed13d105).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22300275
--- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -78,8 +78,8 @@ class DAGSchedulerSuite extends TestKit(ActorSystem("DAGSchedulerSuite")) with F
val taskScheduler = new TaskScheduler() {
override def rootPool: Pool = null
override def schedulingMode: SchedulingMode = SchedulingMode.NONE
- override def start() = {}
- override def stop() = {}
+ override def start() = { started.compareAndSet(false, true) }
--- End diff --
`compareAndSet` is not needed here. Just `set`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69716974
@srowen I've updated this PR and resolved conflict. Please review again. Thanks.
I explain three points:
1. I am not sure the description makes a case that it's significant enough to bother...
Let me give two examples:
(1) When I entered ./bin/spark-sql in command line with yarn-client mode and these resources requests as follows
spark.executor.instances 100
spark.executor.memory 4g
spark.executor.cores 1.
However, I didn't enter sql query string immediately. Because I was interrupted for example I was called to attend a important meeting or I go to fire fighting in our cluster. Even sometimes I forgot enter sql query string.
Then this application ran a night using 100 * 4g * 12h memory resources and 100 * 1 * 12h core resources. But it did nothing.
(2) When SparkContext with 100 spark.executor.instances、4g spark.executor.memory、1 spark.executor.cores was initialized and HadoopRDD scanned 11596 files taking 29.253s to compute splits. And then this job was submitted by DAGScheduler. The resources of 100 * 4g * 29s memory resources and 100 * 1 * 29s core resources were idle.
2. There are several new API methods and changes here.
SparkContext firstly gets applicationId from taskScheduler and uses it to initialize blockManager and eventLogger. And then dagScheduler runs job and submits resources requests to cluster master.
Getting applicationId and submitting resources requests to cluster master are split into two methods.
3. My overall impression is that this adds different code paths and behaviors in different modes for little gain.
I'm sorry that I couldn't get mesos apis to split getting applicationId and submitting resources requests to cluster master into two methods.
Thus slow start of application is currently only supported in YARN mode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68422581
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24934/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69698734
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25457/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68137843
[Test build #24833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24833/consoleFull) for PR 3810 at commit [`05469de`](https://github.com/apache/spark/commit/05469de9f0482bce54a60161b9cb386a64173826).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69562036
[Test build #25413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25413/consoleFull) for PR 3810 at commit [`bf18f5f`](https://github.com/apache/spark/commit/bf18f5f67937348ffc55370b0bf0bc6a62adb063).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69687035
[Test build #25446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25446/consoleFull) for PR 3810 at commit [`3875b1b`](https://github.com/apache/spark/commit/3875b1b0a3120af156c7acf4f103d3edd94b2cb8).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69695646
[Test build #25457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25457/consoleFull) for PR 3810 at commit [`42e02a7`](https://github.com/apache/spark/commit/42e02a7786ab31f4fbd945e481144c89abe53527).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69686713
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25444/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68420005
[Test build #24931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24931/consoleFull) for PR 3810 at commit [`722bd65`](https://github.com/apache/spark/commit/722bd65c425b63c64c2b5f6e736842e66258f27d).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69686710
[Test build #25444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25444/consoleFull) for PR 3810 at commit [`a3954f2`](https://github.com/apache/spark/commit/a3954f26302061ae074a4679fb044d37b1689ad0).
* This patch **fails to build**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68422580
[Test build #24934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24934/consoleFull) for PR 3810 at commit [`18d4f33`](https://github.com/apache/spark/commit/18d4f336703c6d611a8bd30f589de9f424d1f71b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-136915288
@YanTangZhai would you mind closing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69561948
[Test build #25413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25413/consoleFull) for PR 3810 at commit [`bf18f5f`](https://github.com/apache/spark/commit/bf18f5f67937348ffc55370b0bf0bc6a62adb063).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69562038
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25413/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68423000
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24931/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69689372
[Test build #25451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25451/consoleFull) for PR 3810 at commit [`a5d6176`](https://github.com/apache/spark/commit/a5d61764924f74c84fb7c35701dee78c44ddf505).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69695938
[Test build #25458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25458/consoleFull) for PR 3810 at commit [`74094af`](https://github.com/apache/spark/commit/74094afa1bafbc55540f334c3d37fcb3a2570937).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22776371
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -333,9 +333,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
new SparkException("DAGScheduler cannot be initialized due to %s".format(e.getMessage))
}
- // start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's
- // constructor
- taskScheduler.start()
+ if (conf.getBoolean("spark.scheduler.app.slowstart", false) && master == "yarn-client") {
--- End diff --
I'm sorry that I couldn't get mesos apis to split getting applicationId and submitting resources requests to cluster master into two methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68420422
[Test build #24934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24934/consoleFull) for PR 3810 at commit [`18d4f33`](https://github.com/apache/spark/commit/18d4f336703c6d611a8bd30f589de9f424d1f71b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68140334
[Test build #24833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24833/consoleFull) for PR 3810 at commit [`05469de`](https://github.com/apache/spark/commit/05469de9f0482bce54a60161b9cb386a64173826).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai closed the pull request at:
https://github.com/apache/spark/pull/3810
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69698968
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25458/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69698728
[Test build #25457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25457/consoleFull) for PR 3810 at commit [`42e02a7`](https://github.com/apache/spark/commit/42e02a7786ab31f4fbd945e481144c89abe53527).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22776305
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -55,13 +57,9 @@ private[spark] class Client(
* ------------------------------------------------------------------------------------- */
/**
- * Submit an application running our ApplicationMaster to the ResourceManager.
- *
- * The stable Yarn API provides a convenience method (YarnClient#createApplication) for
- * creating applications and setting up the application submission context. This was not
- * available in the alpha API.
+ * Create an application running our ApplicationMaster to the ResourceManager.
*/
- override def submitApplication(): ApplicationId = {
+ override def createApplication(): ApplicationId = {
--- End diff --
SparkContext firstly gets applicationId from taskScheduler and uses it to initialize blockManager and eventLogger. And then dagScheduler runs job and submits resources requests to cluster master.
Getting applicationId and submitting resources requests to cluster master are split into two methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69704756
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25459/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69704753
[Test build #25459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25459/consoleFull) for PR 3810 at commit [`575d57d`](https://github.com/apache/spark/commit/575d57df46b8814355eacefd7e407f48ffb20dd6).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68422997
[Test build #24931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24931/consoleFull) for PR 3810 at commit [`722bd65`](https://github.com/apache/spark/commit/722bd65c425b63c64c2b5f6e736842e66258f27d).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-68140337
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24833/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69687040
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25446/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22300274
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -333,9 +333,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
new SparkException("DAGScheduler cannot be initialized due to %s".format(e.getMessage))
}
- // start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's
- // constructor
- taskScheduler.start()
+ if (conf.getBoolean("spark.scheduler.app.slowstart", false) && master == "yarn-client") {
--- End diff --
There is a new undocumented system property for this? why only client mode?
My overall impression is that this adds different code paths and behaviors in different modes for little gain. I am not sure the description makes a case that it's significant enough to bother
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69687454
[Test build #25448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25448/consoleFull) for PR 3810 at commit [`a0153e4`](https://github.com/apache/spark/commit/a0153e4af52328296b85fa2d1bd2bd69ed13d105).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69687457
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25448/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69686218
[Test build #25444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25444/consoleFull) for PR 3810 at commit [`a3954f2`](https://github.com/apache/spark/commit/a3954f26302061ae074a4679fb044d37b1689ad0).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69700117
[Test build #25459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25459/consoleFull) for PR 3810 at commit [`575d57d`](https://github.com/apache/spark/commit/575d57df46b8814355eacefd7e407f48ffb20dd6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69587246
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25417/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22300276
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -55,13 +57,9 @@ private[spark] class Client(
* ------------------------------------------------------------------------------------- */
/**
- * Submit an application running our ApplicationMaster to the ResourceManager.
- *
- * The stable Yarn API provides a convenience method (YarnClient#createApplication) for
- * creating applications and setting up the application submission context. This was not
- * available in the alpha API.
+ * Create an application running our ApplicationMaster to the ResourceManager.
*/
- override def submitApplication(): ApplicationId = {
+ override def createApplication(): ApplicationId = {
--- End diff --
There are several new API methods and changes here. I don't think they're explained or motivated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69688937
[Test build #25451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25451/consoleFull) for PR 3810 at commit [`a5d6176`](https://github.com/apache/spark/commit/a5d61764924f74c84fb7c35701dee78c44ddf505).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69569742
[Test build #25417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25417/consoleFull) for PR 3810 at commit [`577614f`](https://github.com/apache/spark/commit/577614f40702d6c2aea0d57bf8b52b744e69def3).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69686615
[Test build #25446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25446/consoleFull) for PR 3810 at commit [`3875b1b`](https://github.com/apache/spark/commit/3875b1b0a3120af156c7acf4f103d3edd94b2cb8).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on a diff in the pull request:
https://github.com/apache/spark/pull/3810#discussion_r22776416
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -333,9 +333,15 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
new SparkException("DAGScheduler cannot be initialized due to %s".format(e.getMessage))
}
- // start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's
- // constructor
- taskScheduler.start()
+ if (conf.getBoolean("spark.scheduler.app.slowstart", false) && master == "yarn-client") {
--- End diff --
I'm sorry that I couldn't get mesos apis to split getting applicationId and submitting resources requests to cluster master into two methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69587238
**[Test build #25417 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25417/consoleFull)** for PR 3810 at commit [`577614f`](https://github.com/apache/spark/commit/577614f40702d6c2aea0d57bf8b52b744e69def3) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3810#issuecomment-69689373
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25451/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org