You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by devldevelopment <gi...@git.apache.org> on 2014/10/28 15:49:09 UTC
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
GitHub user devldevelopment opened a pull request:
https://github.com/apache/spark/pull/2980
[SPARK-3611] [WIP] Show number of cores for each executor in application web UI
Initial approach to determine total number of cores on the ExecutorsPage. Extends the information recorded by the storage utils.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devldevelopment/spark SPARK-3611
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2980.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2980
----
commit 785f80ed1a8f39ae06e6eae4f20486a843fb92ab
Author: Your Name <yo...@example.com>
Date: 2014-10-28T14:42:52Z
Initial approach to display number of cores for a given executor.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19533353
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -56,6 +57,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
<table class={UIUtils.TABLE_CLASS}>
<thead>
<th>Executor ID</th>
+ <th>Cores</th>
--- End diff --
It's confusing. In Yarn mode, it looks like it's `--executor-cores` but actually it's the number of available CPU cores.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19533499
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
val memUsed = status.memUsed
val maxMem = status.maxMem
val diskUsed = status.diskUsed
+ val numCores = status.numCoresUsed
--- End diff --
Looks `numCoresUsed` is always the CPU core number of the driver?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60906521
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22447/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by devldevelopment <gi...@git.apache.org>.
Github user devldevelopment commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60908541
Can someone @andrewor14 maybe help me with this failure? java.lang.RuntimeException: spark-core: Binary compatibility check failed! I have changed the constructor of ExecutorSummaryInfo what else needs to be done?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60898397
[Test build #22447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22447/consoleFull) for PR 2980 at commit [`50a1592`](https://github.com/apache/spark/commit/50a15921576862ae99df5542f2e4d3bb36253d1b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19495364
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
val memUsed = status.memUsed
val maxMem = status.maxMem
val diskUsed = status.diskUsed
+ val numCores=status.numCoresUsed
--- End diff --
spaces before and after `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19535011
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
val memUsed = status.memUsed
val maxMem = status.maxMem
val diskUsed = status.diskUsed
+ val numCores = status.numCoresUsed
--- End diff --
I cannot find a simple way to support all modes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by devldevelopment <gi...@git.apache.org>.
Github user devldevelopment commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60898133
Thanks for the feedback guys, updated with changes. Removed the WIP as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-66398809
Hey @devldevelopment given that there is no simple way to support this across different cluster managers, I would recommend that we close this issue for now. We can reopen it once we figure out a better design.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60813614
[Test build #22368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22368/consoleFull) for PR 2980 at commit [`785f80e`](https://github.com/apache/spark/commit/785f80ed1a8f39ae06e6eae4f20486a843fb92ab).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19503970
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala ---
@@ -168,6 +168,9 @@ class StorageStatus(val blockManagerId: BlockManagerId, val maxMem: Long) {
*/
def numRddBlocksById(rddId: Int): Int = _rddBlocks.get(rddId).map(_.size).getOrElse(0)
+ /** Return the number of CPU cores in this block manager. */
+ def numCoresUsed: Int = Runtime.getRuntime.availableProcessors();
--- End diff --
No semicolon here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60824250
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22368/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19504021
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -56,6 +57,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
<table class={UIUtils.TABLE_CLASS}>
<thead>
<th>Executor ID</th>
+ <th>Number of CPU Cores</th>
--- End diff --
I think this could just be "Cores", for consistency with the other UI tables where we list cores.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60812582
add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19495351
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -39,7 +39,8 @@ private case class ExecutorSummaryInfo(
totalInputBytes: Long,
totalShuffleRead: Long,
totalShuffleWrite: Long,
- maxMemory: Long)
+ maxMemory: Long,
+ numCores:Integer)
--- End diff --
This should just be
`numCores: Int`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60812718
Hey @devldevelopment why is this WIP? Are there more changes you plan to add later?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by devldevelopment <gi...@git.apache.org>.
Github user devldevelopment closed the pull request at:
https://github.com/apache/spark/pull/2980
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60906514
[Test build #22447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22447/consoleFull) for PR 2980 at commit [`50a1592`](https://github.com/apache/spark/commit/50a15921576862ae99df5542f2e4d3bb36253d1b).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...
Posted by devldevelopment <gi...@git.apache.org>.
Github user devldevelopment commented on a diff in the pull request:
https://github.com/apache/spark/pull/2980#discussion_r19534611
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
val memUsed = status.memUsed
val maxMem = status.maxMem
val diskUsed = status.diskUsed
+ val numCores = status.numCoresUsed
--- End diff --
It may be, do you have a most robust approach to get it on a per-executor basis?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60824239
[Test build #22368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22368/consoleFull) for PR 2980 at commit [`785f80e`](https://github.com/apache/spark/commit/785f80ed1a8f39ae06e6eae4f20486a843fb92ab).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VectorTransformer(object):`
* `class Normalizer(VectorTransformer):`
* `class JavaModelWrapper(VectorTransformer):`
* `class StandardScalerModel(JavaModelWrapper):`
* `class StandardScaler(object):`
* `class HashingTF(object):`
* `class IDFModel(JavaModelWrapper):`
* `class IDF(object):`
* `class Word2VecModel(JavaModelWrapper):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3611] [WIP] Show number of cores for ea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2980#issuecomment-60767795
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org