You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by CodingCat <gi...@git.apache.org> on 2015/01/21 02:42:24 UTC

[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

GitHub user CodingCat opened a pull request:

    https://github.com/apache/spark/pull/4129

    [SPARK-5337] respect spark.task.cpus when scheduling Applications

    https://issues.apache.org/jira/browse/SPARK-5337
    
    Currently, we didn't 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/CodingCat/spark SPARK-5337

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4129
    
----
commit d383b445bb454d7c8be86bb335792ccab3c368bc
Author: CodingCat <zh...@gmail.com>
Date:   2015-01-21T01:41:45Z

    respect spark.task.cpus when scheduling Applications

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23357881
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
       var registrationDone = false
     
       val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = {
    +    val corePerTask = conf.getInt("spark.task.cpus", 1)
    +    if (corePerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $corePerTask")
    +    }
    +    corePerTask
    +  }
    --- End diff --
    
    Is `corePerTask ` necessary? Why don't just assign `conf.getInt("spark.task.cpus", 1)` to `coreNumPerTask `? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70802269
  
    What if app.coresLeft < app.desc.coreNumPerTask?
    And should we fix the same issue in mesos mode in this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-74147859
  
      [Test build #27373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27373/consoleFull) for   PR 4129 at commit [`43852cb`](https://github.com/apache/spark/commit/43852cb7b61d66d8c611674559869f7d2b4a94ae).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23378931
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
       var registrationDone = false
     
       val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = {
    +    val corePerTask = conf.getInt("spark.task.cpus", 1)
    +    if (corePerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $corePerTask")
    +    }
    +    corePerTask
    +  }
    --- End diff --
    
    >    val coreNumPerTask = conf.getInt("spark.task.cpus", 1)
        if (coreNumPerTask < 1) {
          throw new IllegalArgumentException(
            s"spark.task.cpus is set to an invalid value $coreNumPerTask ")
        }
    
    Is it a better way?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94589740
  
    @andrewor14 , I updated the patch, how about the current version?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299862
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
    @@ -87,7 +89,12 @@ private[spark] class CoarseMesosSchedulerBackend(
     
       override def start() {
         super.start()
    -
    +    
    +    if (coreNumPerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $coreNumPerTask")
    +    }
    --- End diff --
    
    we can do that...but is it a good practice to throw exceptions in the constructor?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23352167
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
       var registrationDone = false
     
       val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = {
    +    val corePerTask = conf.getInt("spark.task.cpus", 1)
    +    if (corePerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $corePerTask")
    +    }
    +    corePerTask
    +  }
    --- End diff --
    
    ah? what do you mean?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70883545
  
    Hi, @WangTaoTheTonic, I just addressed your comments
    
    any other comments?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70897413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25893/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23300002
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -78,7 +78,11 @@ private[spark] class SparkDeploySchedulerBackend(
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
           appUIAddress, sc.eventLogDir)
    -
    +    appDesc.coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    --- End diff --
    
    Eh...seems like I didn't make it clear. I mean in `SparkDeploySchedulerBackend` using a `val` to represent `conf.getInt("spark.task.cpus", 1)` is better. Like:
    
    > val coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    if ( coreNumPerTask  < 1 ) {
       throw ....
    } else {
      appDesc.....
    }


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-79109975
  
    I don't see fine grained mode (MesosSchedulerBackened) being changed at all?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94548522
  
      [Test build #30601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30601/consoleFull) for   PR 4129 at commit [`da8d446`](https://github.com/apache/spark/commit/da8d4468e6789c785b0665655c8e0e3851826b75).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388979
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -556,17 +556,18 @@ private[spark] class Master(
         // in the queue, then the second app, etc.
         if (spreadOutApps) {
           // Try to spread out each app among all the nodes, until it has all its cores
    -      for (app <- waitingApps if app.coresLeft > 0) {
    +      for (app <- waitingApps if app.coresLeft >= app.desc.coreNumPerTask) {
    +        val coreNumPerTask = app.desc.coreNumPerTask
             val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
               .filter(canUse(app, _)).sortBy(_.coresFree).reverse
             val numUsable = usableWorkers.length
             val assigned = new Array[Int](numUsable) // Number of cores to give on each node
             var toAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
             var pos = 0
    -        while (toAssign > 0) {
    -          if (usableWorkers(pos).coresFree - assigned(pos) > 0) {
    -            toAssign -= 1
    -            assigned(pos) += 1
    +        while (toAssign >= coreNumPerTask) {
    --- End diff --
    
    We should add a comment here:
    ```
    // Take into account the number of cores the application uses per task
    // to avoid starving executors or wasting cluster resources (SPARK-5337)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-93317280
  
    sure, will work on it soon


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70883568
  
      [Test build #25892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25892/consoleFull) for   PR 4129 at commit [`d54c5f5`](https://github.com/apache/spark/commit/d54c5f5caca7e3d224656759e394ef6072121202).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78555832
  
      [Test build #28528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28528/consoleFull) for   PR 4129 at commit [`6d76f12`](https://github.com/apache/spark/commit/6d76f123cb239d7670ab59db3474e6bb82a2cfbd).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299588
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -78,7 +78,11 @@ private[spark] class SparkDeploySchedulerBackend(
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
           appUIAddress, sc.eventLogDir)
    -
    +    appDesc.coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    +    if (appDesc.coreNumPerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value ${appDesc.coreNumPerTask}")  
    +    }
    --- End diff --
    
    I'd like do this before assignation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388533
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -533,7 +533,7 @@ private[spark] class Master(
         val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
         val numWorkersAlive = shuffledAliveWorkers.size
         var curPos = 0
    -
    +    
    --- End diff --
    
    please revert


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94554636
  
      [Test build #30603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30603/consoleFull) for   PR 4129 at commit [`c10f980`](https://github.com/apache/spark/commit/c10f980f437d326cab031b6927927c848cdec4cc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23368844
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
       var registrationDone = false
     
       val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = {
    +    val corePerTask = conf.getInt("spark.task.cpus", 1)
    +    if (corePerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $corePerTask")
    +    }
    +    corePerTask
    +  }
    --- End diff --
    
    oh...I tried to embed the validation logic into the assignment, so ...it looks like this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70833836
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25888/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78946474
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28560/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71118479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25975/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70833826
  
      [Test build #25888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25888/consoleFull) for   PR 4129 at commit [`619c8b9`](https://github.com/apache/spark/commit/619c8b9c08ed5f901fc0791644314e3cfd785858).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70895769
  
      [Test build #25892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25892/consoleFull) for   PR 4129 at commit [`d54c5f5`](https://github.com/apache/spark/commit/d54c5f5caca7e3d224656759e394ef6072121202).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78589816
  
      [Test build #28528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28528/consoleFull) for   PR 4129 at commit [`6d76f12`](https://github.com/apache/spark/commit/6d76f123cb239d7670ab59db3474e6bb82a2cfbd).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388951
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -556,17 +556,18 @@ private[spark] class Master(
         // in the queue, then the second app, etc.
         if (spreadOutApps) {
           // Try to spread out each app among all the nodes, until it has all its cores
    -      for (app <- waitingApps if app.coresLeft > 0) {
    +      for (app <- waitingApps if app.coresLeft >= app.desc.coreNumPerTask) {
    --- End diff --
    
    We should add a comment here:
    ```
    // Take into account the number of cores the application uses per task (SPARK-5337)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70897401
  
      [Test build #25893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25893/consoleFull) for   PR 4129 at commit [`9f2a8be`](https://github.com/apache/spark/commit/9f2a8be133421d23c89bbad8c3106787c6e04c39).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388427
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -41,8 +41,20 @@ private[spark] class SparkDeploySchedulerBackend(
     
       private val registrationBarrier = new Semaphore(0)
     
    -  private val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    -  private val totalExpectedCores = maxCores.getOrElse(0)
    +  val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    --- End diff --
    
    to reduce duplicate code, you can get this from `scheduler.CPUS_PER_TASK`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78918114
  
      [Test build #28560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28560/consoleFull) for   PR 4129 at commit [`744e91b`](https://github.com/apache/spark/commit/744e91b954186cde3ce624f1c11beba2493dd1be).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70777339
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25864/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94548787
  
      [Test build #30601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30601/consoleFull) for   PR 4129 at commit [`da8d446`](https://github.com/apache/spark/commit/da8d4468e6789c785b0665655c8e0e3851826b75).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public class MapConfigProvider extends ConfigProvider `
    
     * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-137013558
  
    sure, let me do it in the weekend


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299503
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -78,7 +78,11 @@ private[spark] class SparkDeploySchedulerBackend(
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
           appUIAddress, sc.eventLogDir)
    -
    +    appDesc.coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    --- End diff --
    
    I believe use an `val ` to represent `spark.task.cpus` is better, like `val maxCores = conf.getOption("spark.cores.max").map(_.toInt)`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94574230
  
      [Test build #30602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30602/consoleFull) for   PR 4129 at commit [`55d9143`](https://github.com/apache/spark/commit/55d9143629cc9cb74ae1c1997aba7695a8e7204d).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.
     * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23351906
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -40,6 +40,14 @@ private[spark] class SparkDeploySchedulerBackend(
       var registrationDone = false
     
       val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = {
    +    val corePerTask = conf.getInt("spark.task.cpus", 1)
    +    if (corePerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $corePerTask")
    +    }
    +    corePerTask
    +  }
    --- End diff --
    
    Seems `corePerTask` and  `coreNumPerTask` have some redundancies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70776735
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25863/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70895780
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25892/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-136917833
  
    @CodingCat there have been large changes to the standalone scheduler in 1.5. I don't think this patch in its current state can be easily merged anymore. If you have time, would you mind closing this and reopening a new patch against the latest master?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94577188
  
      [Test build #30603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30603/consoleFull) for   PR 4129 at commit [`c10f980`](https://github.com/apache/spark/commit/c10f980f437d326cab031b6927927c848cdec4cc).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public class MapConfigProvider extends ConfigProvider `
    
     * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388840
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
    @@ -213,13 +225,13 @@ private[spark] class CoarseMesosSchedulerBackend(
             val slaveId = offer.getSlaveId.toString
             val mem = getResource(offer.getResourcesList, "mem")
             val cpus = getResource(offer.getResourcesList, "cpus").toInt
    +        val cpusToUse = math.min(cpus, maxCores - totalCoresAcquired)
    --- End diff --
    
    This doesn't handle your second case right? If `cpusToUse` is 5 and `spark.task.cpus` is 2, then there will be 1 core wasted on the executor. Do we need another check?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70825694
  
      [Test build #25888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25888/consoleFull) for   PR 4129 at commit [`619c8b9`](https://github.com/apache/spark/commit/619c8b9c08ed5f901fc0791644314e3cfd785858).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23300122
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -78,7 +78,11 @@ private[spark] class SparkDeploySchedulerBackend(
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
           appUIAddress, sc.eventLogDir)
    -
    +    appDesc.coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    --- End diff --
    
    I see... will change...thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23300256
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
    @@ -87,7 +89,12 @@ private[spark] class CoarseMesosSchedulerBackend(
     
       override def start() {
         super.start()
    -
    +    
    +    if (coreNumPerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $coreNumPerTask")
    +    }
    --- End diff --
    
    nvm, we don't need to care too much about the exceptions from the constructor ... I will move it to the constructor


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388461
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -41,8 +41,20 @@ private[spark] class SparkDeploySchedulerBackend(
     
       private val registrationBarrier = new Semaphore(0)
     
    -  private val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    -  private val totalExpectedCores = maxCores.getOrElse(0)
    +  val maxCores = conf.getOption("spark.cores.max").map(_.toInt)
    +  val coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    +    
    +  if (coreNumPerTask < 1) {
    +    throw new IllegalArgumentException(
    +      s"spark.task.cpus is set to an invalid value $coreNumPerTask")
    +  }
    --- End diff --
    
    The right place to do this is in `TaskSchedulerImpl`. Also we should be more specific in the error message what "invalid value" means:
    ```
    "spark.task.cpus" must be greater than 0! (was $coresPerTask)
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299355
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -567,10 +567,12 @@ private[spark] class Master(
         } else {
           // Pack each app into as few nodes as possible until we've assigned all its cores
           for (worker <- workers if worker.coresFree > 0 && worker.state == WorkerState.ALIVE) {
    -        for (app <- waitingApps if app.coresLeft > 0) {
    +        for (app <- waitingApps if app.coresLeft >= app.desc.coreNumPerTask) {
               if (canUse(app, worker)) {
    -            val coresToUse = math.min(worker.coresFree, app.coresLeft)
    -            if (coresToUse > 0) {
    +            val coresToUse = math.min(
    +              worker.coresFree - (worker.coresFree % app.desc.coreNumPerTask), 
    +              app.coresLeft - (app.coresLeft % app.desc.coreNumPerTask))
    +            if (coresToUse >= app.desc.coreNumPerTask) {
                   val exec = app.addExecutor(worker, coresToUse)
    --- End diff --
    
    Also too many `app.desc.coreNumPerTask`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-74161462
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27373/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70771251
  
      [Test build #25863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25863/consoleFull) for   PR 4129 at commit [`e28e19f`](https://github.com/apache/spark/commit/e28e19f810c408ab387c03c6307f29fe48b327f7).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70771684
  
      [Test build #25864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25864/consoleFull) for   PR 4129 at commit [`523b3b7`](https://github.com/apache/spark/commit/523b3b7615e8cc86b5428531223ac6b57fd26978).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-85229133
  
    @CodingCat sorry you're right, I didn't realize CPUS_PER_TASK was configured to that flag. LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat closed the pull request at:

    https://github.com/apache/spark/pull/4129


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299327
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -541,17 +541,17 @@ private[spark] class Master(
         // in the queue, then the second app, etc.
         if (spreadOutApps) {
           // Try to spread out each app among all the nodes, until it has all its cores
    -      for (app <- waitingApps if app.coresLeft > 0) {
    +      for (app <- waitingApps if app.coresLeft >= app.desc.coreNumPerTask) {
             val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
               .filter(canUse(app, _)).sortBy(_.coresFree).reverse
             val numUsable = usableWorkers.length
             val assigned = new Array[Int](numUsable) // Number of cores to give on each node
             var toAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
             var pos = 0
    -        while (toAssign > 0) {
    -          if (usableWorkers(pos).coresFree - assigned(pos) > 0) {
    -            toAssign -= 1
    -            assigned(pos) += 1
    +        while (toAssign >= app.desc.coreNumPerTask) {
    +          if (usableWorkers(pos).coresFree - assigned(pos) >= app.desc.coreNumPerTask) {
    +            toAssign -= app.desc.coreNumPerTask
    +            assigned(pos) += app.desc.coreNumPerTask
    --- End diff --
    
    Too many `app.desc.coreNumPerTask` here. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-93155407
  
    @CodingCat good to see this being fixed. However, as it stands the existing solution does not seem to be sufficient. I pointed out a scenario where we will fall into an infinite scheduling loop if the conditions are not met. Also, once you have the time would you mind bringing this up to date? I believe your other patch I just merged in today conflicts quite significantly with your changes here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70770867
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25862/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94574254
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30602/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28388523
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala ---
    @@ -30,6 +30,8 @@ private[spark] class ApplicationDescription(
     
       val user = System.getProperty("user.name", "<unknown>")
     
    +  var coreNumPerTask = 1
    --- End diff --
    
    This field is somewhat arbitrary. A better place to put it would be in the constructor. Also I would rename it to be `coresPerTask`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78946443
  
      [Test build #28560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28560/consoleFull) for   PR 4129 at commit [`744e91b`](https://github.com/apache/spark/commit/744e91b954186cde3ce624f1c11beba2493dd1be).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71117106
  
      [Test build #25974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25974/consoleFull) for   PR 4129 at commit [`bca1080`](https://github.com/apache/spark/commit/bca108099a239ac3f9d47b6e876f40e52bf214f7).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299757
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
    @@ -87,7 +89,12 @@ private[spark] class CoarseMesosSchedulerBackend(
     
       override def start() {
         super.start()
    -
    +    
    +    if (coreNumPerTask < 1) {
    +      throw new IllegalArgumentException(
    +        s"spark.task.cpus is set to an invalid value $coreNumPerTask")
    +    }
    --- End diff --
    
    Should we put the `if` statement together with its initialization?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70776731
  
      [Test build #25863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25863/consoleFull) for   PR 4129 at commit [`e28e19f`](https://github.com/apache/spark/commit/e28e19f810c408ab387c03c6307f29fe48b327f7).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70825357
  
    just upload the fix for Mesos...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r24713483
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
    @@ -213,13 +225,13 @@ private[spark] class CoarseMesosSchedulerBackend(
             val slaveId = offer.getSlaveId.toString
             val mem = getResource(offer.getResourcesList, "mem")
             val cpus = getResource(offer.getResourcesList, "cpus").toInt
    +        val cpusToUse = math.min(cpus, maxCores - totalCoresAcquired)
             if (totalCoresAcquired < maxCores &&
    +            cpusToUse >= coreNumPerTask &&
    --- End diff --
    
    How about fine grained mode in Mesos?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-79120089
  
    If I read the code correctly , I don't need to change that, as it starts mesos task with the correct cpu setup https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L142


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70777331
  
      [Test build #25864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25864/consoleFull) for   PR 4129 at commit [`523b3b7`](https://github.com/apache/spark/commit/523b3b7615e8cc86b5428531223ac6b57fd26978).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78888313
  
    Looks good logically. But we need to do some tests based on this commit especially in mesos mode(fine/coarse grained).
    
    @pwendell @JoshRosen Any comments?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94551829
  
      [Test build #30602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30602/consoleFull) for   PR 4129 at commit [`55d9143`](https://github.com/apache/spark/commit/55d9143629cc9cb74ae1c1997aba7695a8e7204d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94548790
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30601/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70823105
  
    Hi, @WangTaoTheTonic , if app.coresLeft < app.desc.coreNumPerTask, we should not assign more cores to the executor...as those cores will just be wasted since we cannot run a task with that...
    
    in the PR, it's just
    
    ```
     while (toAssign >= app.desc.coreNumPerTask) {
    
    ```
    
    and 
    
    ```
    if (coresToUse >= app.desc.coreNumPerTask) {
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71117113
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25974/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71031642
  
    Besides, the # of cores needed by an application is set via `spark.cores.max`.
    
    What will happen if we set `spark.cores.max` < `spark.task.cpus` ? I think the appcalition submitted will hang up infinitely.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r28389067
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -556,17 +556,18 @@ private[spark] class Master(
         // in the queue, then the second app, etc.
         if (spreadOutApps) {
           // Try to spread out each app among all the nodes, until it has all its cores
    -      for (app <- waitingApps if app.coresLeft > 0) {
    +      for (app <- waitingApps if app.coresLeft >= app.desc.coreNumPerTask) {
    +        val coreNumPerTask = app.desc.coreNumPerTask
             val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
               .filter(canUse(app, _)).sortBy(_.coresFree).reverse
             val numUsable = usableWorkers.length
             val assigned = new Array[Int](numUsable) // Number of cores to give on each node
             var toAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
             var pos = 0
    -        while (toAssign > 0) {
    -          if (usableWorkers(pos).coresFree - assigned(pos) > 0) {
    -            toAssign -= 1
    -            assigned(pos) += 1
    +        while (toAssign >= coreNumPerTask) {
    +          if (usableWorkers(pos).coresFree - assigned(pos) >= coreNumPerTask) {
    +            toAssign -= coreNumPerTask
    +            assigned(pos) += coreNumPerTask
    --- End diff --
    
    is it possible for this to fall into an infinite loop? Let's say I have 5 workers, each of them has 1 core left. The application's `spark.task.cpus` is 2. Here we will never assign any worker resources because not one worker has enough resources for the application.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70770783
  
      [Test build #25862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25862/consoleFull) for   PR 4129 at commit [`d383b44`](https://github.com/apache/spark/commit/d383b445bb454d7c8be86bb335792ccab3c368bc).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70770864
  
      [Test build #25862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25862/consoleFull) for   PR 4129 at commit [`d383b44`](https://github.com/apache/spark/commit/d383b445bb454d7c8be86bb335792ccab3c368bc).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71108212
  
      [Test build #25975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25975/consoleFull) for   PR 4129 at commit [`8b088b2`](https://github.com/apache/spark/commit/8b088b26cd830a9c9872165151e2454dd8abc3b4).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-94577204
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30603/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71118466
  
      [Test build #25975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25975/consoleFull) for   PR 4129 at commit [`8b088b2`](https://github.com/apache/spark/commit/8b088b26cd830a9c9872165151e2454dd8abc3b4).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-74161445
  
      [Test build #27373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27373/consoleFull) for   PR 4129 at commit [`43852cb`](https://github.com/apache/spark/commit/43852cb7b61d66d8c611674559869f7d2b4a94ae).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78555056
  
    @WangTaoTheTonic any other comments?
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r26372564
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -85,8 +97,8 @@ private[spark] class SparkDeploySchedulerBackend(
           args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
    -      appUIAddress, sc.eventLogDir, sc.eventLogCodec)
    -
    +      appUIAddress, sc.eventLogDir)
    --- End diff --
    
    Why remove `sc.eventLogCodec`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-78589839
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28528/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4129#discussion_r23299681
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala ---
    @@ -78,7 +78,11 @@ private[spark] class SparkDeploySchedulerBackend(
         val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
         val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
           appUIAddress, sc.eventLogDir)
    -
    +    appDesc.coreNumPerTask = conf.getInt("spark.task.cpus", 1)
    --- End diff --
    
    I chose var with the concern that, if I use val, that means I need to add it as the constructor parameter...just not sure if I can pass the binary compatibility checking 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-70884543
  
      [Test build #25893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25893/consoleFull) for   PR 4129 at commit [`9f2a8be`](https://github.com/apache/spark/commit/9f2a8be133421d23c89bbad8c3106787c6e04c39).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/4129#issuecomment-71106555
  
      [Test build #25974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25974/consoleFull) for   PR 4129 at commit [`bca1080`](https://github.com/apache/spark/commit/bca108099a239ac3f9d47b6e876f40e52bf214f7).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org