You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hellertime <gi...@git.apache.org> on 2016/05/05 13:34:48 UTC
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
GitHub user hellertime opened a pull request:
https://github.com/apache/spark/pull/12933
[Spark-15155][Mesos] Selectively accept Mesos resources by role
Add new property `spark.mesos.acceptedResourceRoles`. When set, Spark will only accept resources with roles that match. When unset, Spark operates as is, accepting resources from `*` and `spark.mesos.role` if set.
## How was this patch tested?
Additional unit tests added to `MesosSchedulerBackendSuite`, extending the original multi-role test suite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hellertime/spark SPARK-15155
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12933.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12933
----
commit ec4a09667354ee5582bde35098499a9e26776a89
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-04T11:58:55Z
Limit resources to accepted roles
commit 83bc40d9b21022470e5f4b0473bc033f79aabfb1
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-04T12:53:12Z
Move defaultAcceptedRoles logic into Util
commit 66f7822fe7e5a9ad8eeef49d92e436548f46e2a6
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T11:45:13Z
Pre-filter resources
commit 4d2941ebab68b6380a52cd454ccd49bd4077f81b
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T11:52:07Z
Convert to scala list to use filter
commit fda6c713cdb66efd1fa1166168dfcc5620230ec4
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T12:59:36Z
Rework utility function
commit 86516c66777d5dbde377ef37d0d30e207de0fa21
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T13:00:14Z
Update docs
commit 77c0685b7a30f2a4d5f87794b30a683676571919
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T13:26:21Z
Remove double restrict
commit 48b472329036e62ef00973f912d54db7fc5d7872
Author: Chris Heller <he...@gmail.com>
Date: 2016-05-05T13:26:30Z
Add testsuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/12933
Hi @hellertime, this PR seems inactive for few months after the last review comments above. Would this be better closed if you are not currently able to work on this further maybe?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the issue:
https://github.com/apache/spark/pull/12933
Fixed scala style error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12933
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217446062
**[Test build #57993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57993/consoleFull)** for PR 12933 at commit [`d2b7ad4`](https://github.com/apache/spark/commit/d2b7ad444e02b947f4a7264018b4e48610731408).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66740/consoleFull)** for PR 12933 at commit [`e81ff73`](https://github.com/apache/spark/commit/e81ff73a53eeb7eed645f3042ae9da758a254179).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
@hellertime Are you able to rebase?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167812
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57891/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
@srowen can you help review this? Besides my minor comment overall it looks fine with me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
I just tried running it locally and I'm getting the same error. It seems like with your change that test is simply declining the offer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217445754
Rebasing against master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12933#discussion_r91195201
--- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala ---
@@ -50,6 +50,44 @@ trait MesosSchedulerUtils extends Logging {
protected var mesosDriver: SchedulerDriver = null
/**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param conf Spark configuration
+ */
+ protected def getAcceptedResourceRoles(conf: SparkConf): Set[String] = {
+ getAcceptedResourceRoles(
+ conf.getBoolean("spark.mesos.ignoreDefaultRoleResources", false),
+ conf.getOption("spark.mesos.role"))
+ }
+ /**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param props Mesos driver description schedulerProperties map
+ */
+ protected def getAcceptedResourceRoles(props: Map[String, String]): Set[String] = {
+ getAcceptedResourceRoles(
+ props.get("spark.mesos.ignoreDefaultRoleResources") match {
+ case Some(truth) => truth.toBoolean
+ case None => false
+ },
+ props.get("spark.mesos.role"))
+ }
+ /**
+ * Internal version of getAcceptedResourceRoles
+ * @param ignoreDefaultRoleResources user specified property
+ * @param role user specified property
+ */
+ private def getAcceptedResourceRoles(
+ ignoreDefaultRoleResources: Boolean,
+ role: Option[String]) = {
+ val roles = ignoreDefaultRoleResources match {
+ case true if role.isDefined => Set(role)
+ case _ => Set(Some("*"), role)
+ }
+ val acceptedRoles = roles.flatten
+ logDebug(s"Accepting resources from role(s): ${acceptedRoles.mkString(",")}")
--- End diff --
I think we should move this log outside of this helper method, as there might be other context in the future that's calling this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167410
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66887/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66811/consoleFull)** for PR 12933 at commit [`bbe908e`](https://github.com/apache/spark/commit/bbe908ef66898624166bcf06d1773008e7414f14).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217156405
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217231481
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66780/consoleFull)** for PR 12933 at commit [`838dc77`](https://github.com/apache/spark/commit/838dc77d5473e7a584efbd3ac223eba696a427f7).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217231484
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57905/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217441761
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57989/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217341428
True. So really the desired behavior is always just to ignore `*` resources. A boolean property would suffice here. What about `spark.mesos.ignoreDefaultResourceRole`, and have the property ignored if `spark.mesos.role` is unset?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217423623
Alright, I've modified the code to convert from a property which enumerated the accepted roles, to one which will simply ignore the default role when it is set.
I also removed one unit test, which was redundant with the test that checks we accepts all roles when `spark.mesos.role` is set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
I ran `mvn test` inside of the mesos folder.
On Thu, Oct 13, 2016 at 3:21 AM, Chris Heller <no...@github.com>
wrote:
> You saw the error with `./dev/run-tests`? Ok I'll figure this out.
>
> Sent from my iPhone
>
> > On Oct 13, 2016, at 12:24 AM, Timothy Chen <no...@github.com>
> wrote:
> >
> > I just tried running it locally and I'm getting the same error. It seems
> like with your change that test is simply declining the offer.
>
> >
> > \u2014
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub, or mute the thread.
> >
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/12933#issuecomment-253474465>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAEGrAU4TKCiLBwMpDm7PmGsEm79tdoEks5qzgY0gaJpZM4IYBAd>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217231450
**[Test build #57905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57905/consoleFull)** for PR 12933 at commit [`8419664`](https://github.com/apache/spark/commit/84196648410ffdf9d51900559abfe0ef83764580).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66887/consoleFull)** for PR 12933 at commit [`d64b261`](https://github.com/apache/spark/commit/d64b26171c013772c680333c8c742e4965df3b1d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the issue:
https://github.com/apache/spark/pull/12933
@tnachen rebased with master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12933#discussion_r91198706
--- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -551,12 +551,16 @@ private[spark] class MesosClusterScheduler(
currentOffers: List[ResourceOffer],
tasks: mutable.HashMap[OfferID, ArrayBuffer[TaskInfo]]): Unit = {
for (submission <- candidates) {
+ val acceptedResourceRoles = getAcceptedResourceRoles(submission.schedulerProperties)
val driverCpu = submission.cores
val driverMem = submission.mem
logTrace(s"Finding offer to launch driver with cpu: $driverCpu, mem: $driverMem")
val offerOption = currentOffers.find { o =>
- getResource(o.resources, "cpus") >= driverCpu &&
- getResource(o.resources, "mem") >= driverMem
+ val acceptableResources = o.resources.asScala
+ .filter((r: Resource) => acceptedResourceRoles(r.getRole))
--- End diff --
Just a style thing, but I think you'd generally just write `(r => acceptedResourceRoles(r.getRole))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12933#discussion_r91199089
--- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala ---
@@ -50,6 +50,44 @@ trait MesosSchedulerUtils extends Logging {
protected var mesosDriver: SchedulerDriver = null
/**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param conf Spark configuration
+ */
+ protected def getAcceptedResourceRoles(conf: SparkConf): Set[String] = {
+ getAcceptedResourceRoles(
+ conf.getBoolean("spark.mesos.ignoreDefaultRoleResources", false),
+ conf.getOption("spark.mesos.role"))
+ }
+ /**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param props Mesos driver description schedulerProperties map
+ */
+ protected def getAcceptedResourceRoles(props: Map[String, String]): Set[String] = {
+ getAcceptedResourceRoles(
+ props.get("spark.mesos.ignoreDefaultRoleResources") match {
--- End diff --
This might be my taste only but I'm finding this style hard to read.
`props.get(...).map(_.toBoolean).getOrElse(false)`? It's what I'd expect to see in the spark code base but don't know if it's objectively any better. Still I'd break this out into a `val` rather than squeeze this into a method arg
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the issue:
https://github.com/apache/spark/pull/12933
Hi! Was busy with the day job. I didn't mean to let this slip! Absolutely will rebase and retest. Thanks.
Sent from my iPhone
> On Jun 21, 2016, at 11:16 PM, Timothy Chen <no...@github.com> wrote:
>
> @hellertime ping
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66780/consoleFull)** for PR 12933 at commit [`838dc77`](https://github.com/apache/spark/commit/838dc77d5473e7a584efbd3ac223eba696a427f7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217472869
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12933#discussion_r91198770
--- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala ---
@@ -551,12 +551,16 @@ private[spark] class MesosClusterScheduler(
currentOffers: List[ResourceOffer],
tasks: mutable.HashMap[OfferID, ArrayBuffer[TaskInfo]]): Unit = {
for (submission <- candidates) {
+ val acceptedResourceRoles = getAcceptedResourceRoles(submission.schedulerProperties)
val driverCpu = submission.cores
val driverMem = submission.mem
logTrace(s"Finding offer to launch driver with cpu: $driverCpu, mem: $driverMem")
val offerOption = currentOffers.find { o =>
- getResource(o.resources, "cpus") >= driverCpu &&
- getResource(o.resources, "mem") >= driverMem
+ val acceptableResources = o.resources.asScala
+ .filter((r: Resource) => acceptedResourceRoles(r.getRole))
+ .asJava
+ getResource(acceptableResources, "cpus") >= driverCpu &&
+ getResource(acceptableResources, "mem") >= driverMem
--- End diff --
Indent this continuation 2 spaces to make it clear it's a continuation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66811/consoleFull)** for PR 12933 at commit [`bbe908e`](https://github.com/apache/spark/commit/bbe908ef66898624166bcf06d1773008e7414f14).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217472670
**[Test build #57993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57993/consoleFull)** for PR 12933 at commit [`d2b7ad4`](https://github.com/apache/spark/commit/d2b7ad444e02b947f4a7264018b4e48610731408).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66740/consoleFull)** for PR 12933 at commit [`e81ff73`](https://github.com/apache/spark/commit/e81ff73a53eeb7eed645f3042ae9da758a254179).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the issue:
https://github.com/apache/spark/pull/12933
You saw the error with `./dev/run-tests`? Ok I'll figure this out.
Sent from my iPhone
> On Oct 13, 2016, at 12:24 AM, Timothy Chen <no...@github.com> wrote:
>
> I just tried running it locally and I'm getting the same error. It seems like with your change that test is simply declining the offer.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
@hellertime Can you rebase and submit again to rerun the tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217337832
I see, I wonder besides the wildcard role if it ever makes sense to have two lists of roles, one set that you are registered with and another set you want to accept resources for, I cannot think of any possible scenario.
Perhaps we should just introduce one boolean flag that default takes wildcard role resources or not? spark.mesos.rejectWildcardRole or something better named?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217339163
I see that as a potential solution, however at some future point mesos will support frameworks which can hold multiple roles (MESOS-1763 😉) so perhaps leaving it in this form will ease that transition (though we'd then need to change the other property to `spark.mesos.roles`, so really its a wash).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66825/consoleFull)** for PR 12933 at commit [`81e07c8`](https://github.com/apache/spark/commit/81e07c876faab4cb9b9327df5ca9749c52e3cdc2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167806
**[Test build #57891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57891/consoleFull)** for PR 12933 at commit [`48b4723`](https://github.com/apache/spark/commit/48b472329036e62ef00973f912d54db7fc5d7872).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217341006
Yes I am aware of multi roles, but not sure in what situation you would want to register with multiple roles but only use one role's resources?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66825/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66825/consoleFull)** for PR 12933 at commit [`81e07c8`](https://github.com/apache/spark/commit/81e07c876faab4cb9b9327df5ca9749c52e3cdc2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167809
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217472873
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57993/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66740/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the issue:
https://github.com/apache/spark/pull/12933
@hellertime ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217441759
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217156017
@tnachen I'd like to request you review this change, since you are the author of the original mesos role code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12933: [Spark-15155][Mesos] Optionally ignore default ro...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12933#discussion_r91199232
--- Diff: mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala ---
@@ -50,6 +50,44 @@ trait MesosSchedulerUtils extends Logging {
protected var mesosDriver: SchedulerDriver = null
/**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param conf Spark configuration
+ */
+ protected def getAcceptedResourceRoles(conf: SparkConf): Set[String] = {
+ getAcceptedResourceRoles(
+ conf.getBoolean("spark.mesos.ignoreDefaultRoleResources", false),
+ conf.getOption("spark.mesos.role"))
+ }
+ /**
+ * Returns the configured set of roles that an offer can be selected from
+ * @param props Mesos driver description schedulerProperties map
+ */
+ protected def getAcceptedResourceRoles(props: Map[String, String]): Set[String] = {
+ getAcceptedResourceRoles(
+ props.get("spark.mesos.ignoreDefaultRoleResources") match {
+ case Some(truth) => truth.toBoolean
+ case None => false
+ },
+ props.get("spark.mesos.role"))
+ }
+ /**
+ * Internal version of getAcceptedResourceRoles
+ * @param ignoreDefaultRoleResources user specified property
+ * @param role user specified property
+ */
+ private def getAcceptedResourceRoles(
+ ignoreDefaultRoleResources: Boolean,
+ role: Option[String]) = {
+ val roles = ignoreDefaultRoleResources match {
--- End diff --
This seems more readable:
```
val roles = if (ignoreDefaultRoleResources && role.isDefined) Set(role) else Set(Some("*"), role)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66780/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66741/consoleFull)** for PR 12933 at commit [`181f207`](https://github.com/apache/spark/commit/181f2075199549571364224f45914831b49f3310).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66741/consoleFull)** for PR 12933 at commit [`181f207`](https://github.com/apache/spark/commit/181f2075199549571364224f45914831b49f3310).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217423509
**[Test build #57989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57989/consoleFull)** for PR 12933 at commit [`5ea2f46`](https://github.com/apache/spark/commit/5ea2f46909db99bed0a78fe3f2dc7af316087c36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217213224
@tnachen exactly, I want to keep Spark from grabbing * roles, as in my use case I have a particular spark cluster that I want to isolate from other clusters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the issue:
https://github.com/apache/spark/pull/12933
@tnachen I'm not sure what to do about this unit test failure. Running `./dev/run-tests` on my system does not produce the error, and trying to just run the mesos suite with `./build/mvn -DwildcardSuites=org.apache.spark.scheduler.cluster.mesos test` results in a successful build, it doesn't appear to actually run the test. Any advice on how to test this locally?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217343655
I'm updating the title of the PR to reflect the change in approach. I think the boolean property will be sufficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167334
So from what I can see, what you really want is to only don't select resources from * role right?
Otherwise if you're getting offers from other roles you are already registered for those roles as well, which begs the question why are you registered with that role as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167540
**[Test build #57891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57891/consoleFull)** for PR 12933 at commit [`48b4723`](https://github.com/apache/spark/commit/48b472329036e62ef00973f912d54db7fc5d7872).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217167363
jenkins please test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12933
**[Test build #66887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66887/consoleFull)** for PR 12933 at commit [`d64b261`](https://github.com/apache/spark/commit/d64b26171c013772c680333c8c742e4965df3b1d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66741/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217229137
**[Test build #57905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57905/consoleFull)** for PR 12933 at commit [`8419664`](https://github.com/apache/spark/commit/84196648410ffdf9d51900559abfe0ef83764580).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Selectively accept Mesos ...
Posted by hellertime <gi...@git.apache.org>.
Github user hellertime commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217240387
*MiMa* test died here:
```
/home/jenkins/workspace/SparkPullRequestBuilder@2/dev/mima: line 37: 18083 Aborted (core dumped) java -XX:MaxPermSize=1g -Xmx2g -cp "$TOOLS_CLASSPATH:$OLD_DEPS_CLASSPATH" org.apache.spark.tools.GenerateMIMAIgnore
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/dev/mima -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive-thriftserver -Phive ; received return code 134
Attempting to post to Github...
> Post successful.
```
I don't think this is due to my change, but I'm really not familiar with the *MiMa* test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66811/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12933
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15155][Mesos] Optionally ignore default...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12933#issuecomment-217441601
**[Test build #57989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57989/consoleFull)** for PR 12933 at commit [`5ea2f46`](https://github.com/apache/spark/commit/5ea2f46909db99bed0a78fe3f2dc7af316087c36).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org