You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Astralidea <gi...@git.apache.org> on 2016/01/15 09:32:10 UTC
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
GitHub user Astralidea opened a pull request:
https://github.com/apache/spark/pull/10768
[SPARK-12832][CORE] Fix dispatcher does not have a constraints config
before offer resouce, first meet meetsConstraints to filter some offer.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Astralidea/spark fix-spark-12832
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10768.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10768
----
commit 6e8a028fce5d9fe6df69fd79e7a2c6ff748a575f
Author: xueyan.li <xu...@qunar.com>
Date: 2016-01-15T08:28:05Z
[SPARK-12832][CORE] Fix dispatcher does not have a constraints config
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172016154
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49460/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172016148
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172207582
@Astralidea I don't understand your question, sorry.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][MESOS] Fix dispatcher does not h...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-178963188
@dragos ok, I hope this issue could fix in next version. only read once spark.mesos.constraints is enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172149501
@dragos A question, should I also change MesosClusterDispatcher.scala code for cmd argument in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172149787
**[Test build #49516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49516/consoleFull)** for PR 10768 at commit [`ef698a9`](https://github.com/apache/spark/commit/ef698a9d87cd5f19240a24a56ae638ac15cf1c0e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172149790
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49516/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/10768#discussion_r50090479
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -142,6 +142,8 @@ private[spark] class MesosClusterScheduler(
private val queuedDriversState = engineFactory.createEngine("driverQueue")
private val launchedDriversState = engineFactory.createEngine("launchedDrivers")
private val pendingRetryDriversState = engineFactory.createEngine("retryList")
+ private val driverOfferConstraints =
+ parseConstraintString(conf.get("spark.mesos.constraints", ""))
--- End diff --
This is the problem: This setting is read only once, when the dispatcher is started. What you want is to pick up the constraints set on the submitted job. Have a look at how other job-specific settings are treated in the code (there's a submitProperties variable, or something similar).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-178178342
FYI #10949 is another patch for the same issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172149788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][MESOS] Fix dispatcher does not h...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-178313347
@dragos Sorry about busy these days, In your comment, The feature is great. Because if I change the configuration I did not redeploy mesos-dispacher, but In my System it is enough to use I didn't change config everyday, it is merely stable. and I did not figure out how to reload configure every time. If I have time to do this I will push another patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/10768#discussion_r49862082
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -142,6 +142,8 @@ private[spark] class MesosClusterScheduler(
private val queuedDriversState = engineFactory.createEngine("driverQueue")
private val launchedDriversState = engineFactory.createEngine("launchedDrivers")
private val pendingRetryDriversState = engineFactory.createEngine("retryList")
+ // dispactcher constraints
--- End diff --
There's a typo there. I think the comment isn't necessary though, the name of the variable is clear enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172149704
**[Test build #49516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49516/consoleFull)** for PR 10768 at commit [`ef698a9`](https://github.com/apache/spark/commit/ef698a9d87cd5f19240a24a56ae638ac15cf1c0e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172510959
Thanks for clarifying. I think this is a bit confusing. You want to respect the Mesos constraints that belong to each particular job that is submitted to the dispatcher. Those configuration options should come from a submission request, not from the Spark config options that are used to launch the dispatcher (probably found inside `schedulerProperties`)
Right now, the code would only launch drivers on one particular set of constraints, defined when the dispatcher is launched. I believe the better solution is to allow each Spark job to define its Mesos constraints independently, when submitting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-171902147
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-171989197
**[Test build #49460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49460/consoleFull)** for PR 10768 at commit [`6e8a028`](https://github.com/apache/spark/commit/6e8a028fce5d9fe6df69fd79e7a2c6ff748a575f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/10768#discussion_r49862201
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -519,6 +521,20 @@ private[spark] class MesosClusterScheduler(
val currentTime = new Date()
stateLock.synchronized {
+ var it = offers.iterator()
+ while (it.hasNext) {
+ val offer = it.next()
+ val offerAttributes = toAttributeMap(offer.getAttributesList)
+ val meetsConstraints = matchesAttributeRequirements(driverOfferConstraints, offerAttributes)
+
+ if (!meetsConstraints) {
+ driver.declineOffer(offer.getId)
+ it.remove()
--- End diff --
I'd rather not mutate the underlying collection. Anyhow, this has no effect because the code below is using `currentOffers`, which is a copy of the original resources made earlier. I suggest you insert this filter at that point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172158013
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][MESOS] Fix dispatcher does not h...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea closed the pull request at:
https://github.com/apache/spark/pull/10768
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172157973
**[Test build #49517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49517/consoleFull)** for PR 10768 at commit [`f4d586f`](https://github.com/apache/spark/commit/f4d586f733de44104171fa2cc050bafd848c7059).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172151231
**[Test build #49517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49517/consoleFull)** for PR 10768 at commit [`f4d586f`](https://github.com/apache/spark/commit/f4d586f733de44104171fa2cc050bafd848c7059).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172723384
@dragos Thank you for your answers. so the code may does not need to change other.
I will make a test when I change conf file spark-defaults.conf. Wheather it will take effect on deploy a driver.That is .when submit a job.will it read spark-defaults.conf every time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172158014
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49517/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-178223014
Also @Astralidea please change the title to include `[MESOS]` instead of `[CORE]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172793571
@Astralidea The code needs to change, but probably the change is minor. See [my comment](https://github.com/apache/spark/pull/10768#discussion_r50090479)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by Astralidea <gi...@git.apache.org>.
Github user Astralidea commented on a diff in the pull request:
https://github.com/apache/spark/pull/10768#discussion_r49925683
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -519,6 +521,20 @@ private[spark] class MesosClusterScheduler(
val currentTime = new Date()
stateLock.synchronized {
+ var it = offers.iterator()
+ while (it.hasNext) {
+ val offer = it.next()
+ val offerAttributes = toAttributeMap(offer.getAttributesList)
+ val meetsConstraints = matchesAttributeRequirements(driverOfferConstraints, offerAttributes)
+
+ if (!meetsConstraints) {
+ driver.declineOffer(offer.getId)
+ it.remove()
--- End diff --
oh, it's true. I modify code base in 1.5.2 previously, and test ok in practice. in 1.6.0 version the code is change. I will revise it soon.thanks for your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172015840
**[Test build #49460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49460/consoleFull)** for PR 10768 at commit [`6e8a028`](https://github.com/apache/spark/commit/6e8a028fce5d9fe6df69fd79e7a2c6ff748a575f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][MESOS] Fix dispatcher does not h...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-178503147
@Astralidea I think we should focus on getting #10949 in, which implements exactly this behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12832][CORE] Fix dispatcher does not ha...
Posted by BrickXu <gi...@git.apache.org>.
Github user BrickXu commented on the pull request:
https://github.com/apache/spark/pull/10768#issuecomment-172525962
@dragos great idea!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org