You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ajbozarth <gi...@git.apache.org> on 2015/12/05 03:47:40 UTC
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
GitHub user ajbozarth opened a pull request:
https://github.com/apache/spark/pull/10154
[SPARK-12149] [Web UI] Executor UI improvement suggestions - Color UI
Added color coding to the Executors page for Active Tasks, Failed Tasks, Completed Tasks and Task Time.
Active Tasks is shaded green with it's range based on percentage of total cores used.
Failed Tasks is shaded red ranging over the first 10% of total tasks failed
Completed Tasks is shaded blue ranging over 10% of total tasks including failed and active tasks.
Task Time is shaded red when GC Time goes over 10% of total time with it's range directly corresponding to the percent of total time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajbozarth/spark spark12149
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10154.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10154
----
commit 71750e4a74ca9d3fa842b186c496aa752b7a13a4
Author: Alex Bozarth <aj...@us.ibm.com>
Date: 2015-12-02T01:19:43Z
SPARK-1832 added color to tasks on executor page
commit d630cac67e441c396a60c9add0e185bbb2cc6b1c
Author: Alex Bozarth <aj...@us.ibm.com>
Date: 2015-12-04T20:42:16Z
SPARK-12149 Cleaned up color UI code, fixed tests, added color UI for GC time
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174613696
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169084460
**[Test build #48776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48776/consoleFull)** for PR 10154 at commit [`34435a5`](https://github.com/apache/spark/commit/34435a5b407c09c8da6356c688878ff7a21e35c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165408796
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47913/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169423120
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169115892
**[Test build #48786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48786/consoleFull)** for PR 10154 at commit [`5e27bf0`](https://github.com/apache/spark/commit/5e27bf017ac336cdaf9d27495816d97b373cdd44).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173078734
retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172961626
I had an implementation a handful of commits ago that only shaded completed when either active or failed was also colored for that row, allowing for quick comparison. What do you think of that option (there are screen shots of it above)? Also if we got rid of coloring completed tasks do we want to switch active back to green or leave it as blue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-167733524
I think green for completed makes sense, and having it always stand out doesn't seem like a particularly bad thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166419895
I didn't do screen shots of no color for completed, overall I am personally a fan of switch the green to completed and making coloring completed conditional on an active or failed task to compare to. If you agree @srowen then I'll push that code change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171811880
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49421/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172147900
**[Test build #49513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49513/consoleFull)** for PR 10154 at commit [`05f957a`](https://github.com/apache/spark/commit/05f957a7274fbbb918c0765d8608a3ee820c1fac).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171197660
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49291/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162262316
Can you post screenshots?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-168800557
Can someone get this to retest? Also @JoshRosen any opinions on the current commit?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174613081
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-168765294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48674/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166459708
**[Test build #48137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48137/consoleFull)** for PR 10154 at commit [`31d6a1e`](https://github.com/apache/spark/commit/31d6a1ecc8f6a860af46897b492f0d5cc58d1219).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172128948
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49488/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172872675
sorry been super busy with other stuff. I'll hopefully try this out today. I want to see what it looks like.
which modes did you test this in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169149738
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171052468
so one concern I have is how will the user know what the colors mean. Some of them are fairly obvious, but the task time and GC color I don't think is since we don't show GC in this table. I definitely think its useful its just making sure it makes sense to the average user. I wonder if we added a hover and displayed gc time or a description of the color. thoughts or other ideas?
Also note I filed the jira on behalf of one user, we should decide what makes sense in the end.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162685462
It must be a false positive or else some problem with how mima got invoked. I don't see any API changes here. If they're "real" false positives you'll need to add an exclusion but it tells you what to add. Let's see what Jenkins says.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172109493
**[Test build #49488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49488/consoleFull)** for PR 10154 at commit [`b7a1c09`](https://github.com/apache/spark/commit/b7a1c09d483a474a023f24f8b54bafc930c34628).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166457128
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172956190
So after seeing this run I'm not sure how useful the Completed Task column shading it. Especially when there aren't any tasks running. I think it draws your attention to that column for no reason. So I think I would like to see the shading on completed tasks removed unless someone has a strong use case for it, thoughts?
I still need to try out the failed and GC time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166499052
**[Test build #48137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48137/consoleFull)** for PR 10154 at commit [`31d6a1e`](https://github.com/apache/spark/commit/31d6a1ecc8f6a860af46897b492f0d5cc58d1219).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10154: [SPARK-12149] [Web UI] Executor UI improvement suggestio...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/10154
So I looked at the UI today and I have to say the color code is *extremely confusing*. On a normal cluster I saw a bunch of reds, which is usually reserved for errors.
(I'm going to leave a comment on the JIRA ticket too)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171168930
Here's a small screen shot of the change (note I changed GC threshold to 1% for this example)
<img width="251" alt="screen shot 2016-01-12 at 9 06 58 pm" src="https://cloud.githubusercontent.com/assets/13952758/12285584/7e9217c6-b970-11e5-891d-ad53751e8686.png">
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166499142
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48137/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-163066981
added the MiMa exclude, please advise if I did it correctly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171847211
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162134143
Few discussion issues to raise:
1. I chose to use opacity (alpha channel) to range the colors, what do people think of this? I felt it was simpler than figuring out a range of rgb.
2. I chose a 10% threshold for the GC color appearing, is that a good threshold or should I change it?
3. What do people think of my choices for determining the color ranges?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171041806
**[Test build #49257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49257/consoleFull)** for PR 10154 at commit [`5bcc298`](https://github.com/apache/spark/commit/5bcc2983a62c8b3c311a3df65c87c35b5ad1f150).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46852150
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
--- End diff --
You are correct, it does not work in this syntax, I tired.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166458236
**[Test build #48136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48136/consoleFull)** for PR 10154 at commit [`190a033`](https://github.com/apache/spark/commit/190a03340e3795f21e3b8886b5f0c231136cba0b).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `public final class LZ4BlockInputStream extends FilterInputStream `\n * `case class Range(`\n * `case class Range(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162212722
**[Test build #2173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2173/consoleFull)** for PR 10154 at commit [`d630cac`](https://github.com/apache/spark/commit/d630cac67e441c396a60c9add0e185bbb2cc6b1c).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162636439
Thanks for the screenshots. The color really 'pops', so my concern is just making sure it doesn't get overwhelming.
I can see using the red to very clearly call attention to failed things, or excessive GC. It also sort of feels write to call attention to active things while they're active. (If green = good and red = bad and blue = neutral, is blue more appropriate for an active task? it could turn out to fail or complete. shrug.)
I wasn't as sure about coloring the completed tasks column therefore, but it's a matter of taste.
I kind of like how this looks, but if anyone else is concerned about the amount of color, I suppose it's coherent to make the text colored instead of the background.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169149741
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48786/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r50698808
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -233,6 +229,69 @@ private[ui] class ExecutorsPage(
</tbody>
</table>
}
+
+ private def taskData(maxTasks: Int,
+ activeTasks: Int,
+ failedTasks: Int,
+ completedTasks: Int,
+ totalTasks: Int,
+ totalDuration: Long,
+ totalGCTime: Long):
+ Seq[Node] = {
+ // Determine Color Opacity from 0.5-1
+ // activeTasks range from 0 to maxTasks
+ val activeTasksAlpha =
+ if (maxTasks > 0) {
+ (activeTasks.toDouble / maxTasks) * 0.5 + 0.5
+ } else {
+ 1
+ }
+ // failedTasks range max at 10% failure, alpha max = 1
+ val failedTasksAlpha =
+ if (totalTasks > 0) {
+ math.min (10 * failedTasks.toDouble / totalTasks, 1) * 0.5 + 0.5
--- End diff --
remove space after min
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172936958
Thanks, I only tested using a standalone cluster, I have Hadoop and Yarn set up on my remote testing cluster so I can test that now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173288616
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49788/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10154
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166418393
And here's screenshots of completed only being colored when comparing to active or failed, examples with both color choices.
Personally I don't know how I feel about this, I think it looks better visually, but I'm not sure about usability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46759338
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -117,6 +119,28 @@ private[ui] class ExecutorsPage(
val maximumMemory = info.maxMemory
val memoryUsed = info.memoryUsed
val diskUsed = info.diskUsed
+
+ // Determine Color Opacity from 0.5-1
+ var activeTasksAlpha = 1.0
+ var failedTasksAlpha = 1.0
+ var completedTasksAlpha = 1.0
+ var totalDurationAlpha = 1.0
+ if (info.totalCores > 0) {
+ // activeTasks range from 0 to all cores
+ activeTasksAlpha = (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
+ }
+ if (info.totalTasks > 0) {
--- End diff --
You can also assign these like
```
val (failedTasksAlpha, completedTasksAlpha) =
if (...) {
(..., ...)
} else {
(1.0, 1.0)
}
```
but whether that's clearer is a matter of taste.
It might be better to cap alpha at 1.0 just for tidiness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171846799
**[Test build #49428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49428/consoleFull)** for PR 10154 at commit [`07ba26a`](https://github.com/apache/spark/commit/07ba26ad45e45030ffd62e06e501b3bb6abe5310).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171409118
I think this is good for now, we have other columns that don't have tool tips and I think the colors with the others are more obvious. We can add them later is users are confused.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162632744
Here's a few screen shots, note that I lowered the threshold and range for the GC Time Color so you can see it easier for this example (1% threshold with color range over first 5% instead of 10% threshold with range over first 50%)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r49772339
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala ---
@@ -62,6 +64,7 @@ class ExecutorsListener(storageStatusListener: StorageStatusListener) extends Sp
override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = synchronized {
val eid = executorAdded.executorId
executorToLogUrls(eid) = executorAdded.executorInfo.logUrlMap
+ executorToTotalCores(eid) = executorAdded.executorInfo.totalCores
--- End diff --
I looked into the differences between that case and this and the values are coming from different places. In this case the totalCores value is given as a param to ExecutorInfo which is called directly by the Standalone (local) and Mesos backends.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172128431
I've made changes to the totals table to include color and match column data with the executors table.
Here's a screen shot. I'll push the update once it passes tests locally.
<img width="1186" alt="screen shot 2016-01-15 at 3 37 43 pm" src="https://cloud.githubusercontent.com/assets/13952758/12368356/8d81acc0-bb9f-11e5-9af4-b9115d393397.png">
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166393950
I'll switch the green and blue and post some screen shot to see the difference.
As for the completed column always being blue:
I could make it conditional so it's only blue if there are either active or failed tasks to compare too, but I feel like that would end up looking odd when some executors completed tasks are colored and some aren't.
Or I could remove the coloring from completed tasks, I'll post some screenshots of this as well before pushing any changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46759346
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
--- End diff --
Can the `style="..."` text go inside the conditional so we don't end up with `style=""` or does that not work in this syntax?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174612665
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171075720
**[Test build #49257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49257/consoleFull)** for PR 10154 at commit [`5bcc298`](https://github.com/apache/spark/commit/5bcc2983a62c8b3c311a3df65c87c35b5ad1f150).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172154738
**[Test build #49513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49513/consoleFull)** for PR 10154 at commit [`05f957a`](https://github.com/apache/spark/commit/05f957a7274fbbb918c0765d8608a3ee820c1fac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174651435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50000/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171197512
**[Test build #49291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49291/consoleFull)** for PR 10154 at commit [`70ab748`](https://github.com/apache/spark/commit/70ab74829d341af63e1d3b05ecc537d99f2243be).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162690421
**[Test build #2178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2178/consoleFull)** for PR 10154 at commit [`2887aca`](https://github.com/apache/spark/commit/2887aca6eedc75deb648a539433e698dedf05080).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174651158
**[Test build #50000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50000/consoleFull)** for PR 10154 at commit [`9293c41`](https://github.com/apache/spark/commit/9293c413d830f3c8918c5e2293b6dff5a931322d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165378870
Jenkins add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46864981
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
+ } else {
+ ""
+ }
+ }>{info.activeTasks}</td>
+ <td style={
+ if (info.failedTasks > 0) {
+ "background:hsla(0, 100%, 50%, " + failedTasksAlpha + ");color:white"
--- End diff --
Oh so you have to express some as % and some as a fraction? that seems odd. If consistency is at all possible that'd be clearer I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166455307
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165408794
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r47274898
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -33,11 +33,13 @@ private[ui] case class ExecutorSummaryInfo(
rddBlocks: Int,
memoryUsed: Long,
diskUsed: Long,
+ totalCores: Int,
--- End diff --
So the comment for this case class says it isn't used anymore - do we really need to update it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166458281
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48136/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173049613
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49727/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174580984
Fixed the little issues, don't know when the space after min got there though. And this also showed me my auto-format in IntelliJ isn't 100% correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46852090
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -117,6 +119,28 @@ private[ui] class ExecutorsPage(
val maximumMemory = info.maxMemory
val memoryUsed = info.memoryUsed
val diskUsed = info.diskUsed
+
+ // Determine Color Opacity from 0.5-1
+ var activeTasksAlpha = 1.0
+ var failedTasksAlpha = 1.0
+ var completedTasksAlpha = 1.0
+ var totalDurationAlpha = 1.0
+ if (info.totalCores > 0) {
+ // activeTasks range from 0 to all cores
+ activeTasksAlpha = (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
+ }
+ if (info.totalTasks > 0) {
--- End diff --
I wanted to do that but the style guide wasn't clear on whether it was allow, I'll update it since it's cleaner. And I'll add the 1.0 cap, I left it off originally because the code was getting busy and hard to read originally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166408671
There's three screenshots of what switching the blue and green would look like, I actually think I like it better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162211929
**[Test build #2173 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2173/consoleFull)** for PR 10154 at commit [`d630cac`](https://github.com/apache/spark/commit/d630cac67e441c396a60c9add0e185bbb2cc6b1c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165278186
Is there a way someone could get the tests to run to check my change? I assumed they would run on my commit, but they didn't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162684785
I ran the MiMa tests locally (I didn't know they didn't run before) and they're failing because I changed api.scala and added two vals to ExecutorSummary. How do I go about fixing the MiMa failure?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171168341
@tgravescs I've added GC Time (in parens) to the Task Time column and added a tool tip explaining the coloring and what GC Time is. I agree these are probably good additions. Do you think I should add tool tips for the Task columns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r49734086
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -114,9 +114,38 @@ private[ui] class ExecutorsPage(
/** Render an HTML row representing an executor */
private def execRow(info: ExecutorSummary, logsExist: Boolean): Seq[Node] = {
+ // When GCTimePercent is edited change ToolTips.TASK_TIME to match
+ val GCTimePercent = 0.1
+
val maximumMemory = info.maxMemory
val memoryUsed = info.memoryUsed
val diskUsed = info.diskUsed
+
+ // Determine Color Opacity from 0.5-1
+ // activeTasks range from 0 to all cores
+ val activeTasksAlpha =
+ if (info.totalCores > 0) {
+ (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
--- End diff --
So this isn't taking into account that a task could use more then one core via the spark.task.cpus configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171169376
**[Test build #49291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49291/consoleFull)** for PR 10154 at commit [`70ab748`](https://github.com/apache/spark/commit/70ab74829d341af63e1d3b05ecc537d99f2243be).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171811862
**[Test build #49421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49421/consoleFull)** for PR 10154 at commit [`4294d3f`](https://github.com/apache/spark/commit/4294d3f4639be49c8031e35b41c8b59ee241b2fc).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ExecutorsListener(storageStatusListener: StorageStatusListener, conf: SparkConf)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172154787
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r49734927
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala ---
@@ -62,6 +64,7 @@ class ExecutorsListener(storageStatusListener: StorageStatusListener) extends Sp
override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = synchronized {
val eid = executorAdded.executorId
executorToLogUrls(eid) = executorAdded.executorInfo.logUrlMap
+ executorToTotalCores(eid) = executorAdded.executorInfo.totalCores
--- End diff --
so this might have issues on other cluster manager other then yarn.. see the discussion at https://github.com/apache/spark/pull/2980
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166442972
I think that having completed stay colored looks fine when failed is colored next to it, the contrasting shades allows you to quickly tell how bad the failed percentage is. But I agree we need other opinions to help decide. I'll commit the changes and update the top description to match.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172113944
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166453110
**[Test build #48135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48135/consoleFull)** for PR 10154 at commit [`1652274`](https://github.com/apache/spark/commit/1652274abf3d335918d3b84e5b874aa4a58e9100).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171197657
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169109703
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48776/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174616238
**[Test build #50000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50000/consoleFull)** for PR 10154 at commit [`9293c41`](https://github.com/apache/spark/commit/9293c413d830f3c8918c5e2293b6dff5a931322d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169419547
Current look (same screen shot from group above)
For those following, is this a change we want? Any objections?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171826778
**[Test build #49428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49428/consoleFull)** for PR 10154 at commit [`07ba26a`](https://github.com/apache/spark/commit/07ba26ad45e45030ffd62e06e501b3bb6abe5310).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r50698826
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -233,6 +229,69 @@ private[ui] class ExecutorsPage(
</tbody>
</table>
}
+
+ private def taskData(maxTasks: Int,
+ activeTasks: Int,
+ failedTasks: Int,
+ completedTasks: Int,
+ totalTasks: Int,
+ totalDuration: Long,
+ totalGCTime: Long):
+ Seq[Node] = {
+ // Determine Color Opacity from 0.5-1
+ // activeTasks range from 0 to maxTasks
+ val activeTasksAlpha =
+ if (maxTasks > 0) {
+ (activeTasks.toDouble / maxTasks) * 0.5 + 0.5
+ } else {
+ 1
+ }
+ // failedTasks range max at 10% failure, alpha max = 1
+ val failedTasksAlpha =
+ if (totalTasks > 0) {
+ math.min (10 * failedTasks.toDouble / totalTasks, 1) * 0.5 + 0.5
+ } else {
+ 1
+ }
+ // totalDuration range from 0 to 50% GC time, alpha max = 1
+ val totalDurationAlpha =
+ if (totalDuration > 0) {
+ math.min (totalGCTime.toDouble / totalDuration + 0.5, 1)
--- End diff --
remove space after min before (
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166280812
My only reservations are: active vs complete being colored green vs blue instead of the other way around, and also coloring the complete column at all, since it would always be colored after starting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172154788
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49513/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166457097
**[Test build #48135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48135/consoleFull)** for PR 10154 at commit [`1652274`](https://github.com/apache/spark/commit/1652274abf3d335918d3b84e5b874aa4a58e9100).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46852254
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
+ } else {
+ ""
+ }
+ }>{info.activeTasks}</td>
+ <td style={
+ if (info.failedTasks > 0) {
+ "background:hsla(0, 100%, 50%, " + failedTasksAlpha + ");color:white"
--- End diff --
That is actually how HSLa is designed and was it seemed to be an area on contention when it was first defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171076007
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49257/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173011586
thanks, I saw the above changes for only showing completed when active or failed but again I'm not sure how useful it is. I think just dropping the color for now makes sense. I think leave the color blue in active.
If we were to keep completed tasks I think it makes more sense to shade vs other executors. The times I can think of wanting to know completed tasks is different is perhaps when that executor is being slow or skewed data. So if it doesn't have as many completed tasks as other executors. I don't think completed vs total on same executor is really that useful. But I think for now we just remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173242605
**[Test build #49788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49788/consoleFull)** for PR 10154 at commit [`2f54a33`](https://github.com/apache/spark/commit/2f54a333f8d6f9250132679a61783550de5fe34e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173049274
**[Test build #49727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49727/consoleFull)** for PR 10154 at commit [`2f54a33`](https://github.com/apache/spark/commit/2f54a333f8d6f9250132679a61783550de5fe34e).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169423122
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48866/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174612667
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49996/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174651434
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46776484
--- Diff: core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json ---
@@ -4,11 +4,13 @@
"rddBlocks" : 8,
"memoryUsed" : 28000128,
"diskUsed" : 0,
+ "totalCores" : 0,
--- End diff --
The expected number of cores is 0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173287271
**[Test build #49788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49788/consoleFull)** for PR 10154 at commit [`2f54a33`](https://github.com/apache/spark/commit/2f54a333f8d6f9250132679a61783550de5fe34e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172111343
With the Totals table PR #10668 (other sub-jira related to this one) now merged, do we want to add the colorizations to that table also?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173236741
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169109699
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10154: [SPARK-12149] [Web UI] Executor UI improvement suggestio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/10154
Yes, I have not noticed the color to be confusing. What type of state are you looking at @rxin? it could be that with enough activity, failures are inevitable, and marking anything > 0 failures as red is a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174534008
A few minor nits, otherwise looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173982110
I haven't forgot about this, just busy, I want to take one last pass over this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166455310
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48134/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165408654
**[Test build #47913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47913/consoleFull)** for PR 10154 at commit [`3208d85`](https://github.com/apache/spark/commit/3208d85653b0d2df0c68bb67856fac1f50740ecf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166455529
**[Test build #48136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48136/consoleFull)** for PR 10154 at commit [`190a033`](https://github.com/apache/spark/commit/190a03340e3795f21e3b8886b5f0c231136cba0b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162686079
**[Test build #2178 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2178/consoleFull)** for PR 10154 at commit [`2887aca`](https://github.com/apache/spark/commit/2887aca6eedc75deb648a539433e698dedf05080).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166458279
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169418977
After reading @holdenk 's comment and using the colors for a while I've changed completed back to be colored when > 0 even if active and failed are zero. I'll re-upload one example screen shot of what it currently looks like
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165381448
**[Test build #47913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47913/consoleFull)** for PR 10154 at commit [`3208d85`](https://github.com/apache/spark/commit/3208d85653b0d2df0c68bb67856fac1f50740ecf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #10154: [SPARK-12149] [Web UI] Executor UI improvement suggestio...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/10154
@rxin The reds only show up on failed tasks or a high gc time, what would you expect from each color? Also this was a while ago (and one of my first prs) so I'm willing to revisit how this was done. If we want to open a new JIRA to revisit it I'll work on it.
(responding here since this is where all the reviewers are looped in)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-162134207
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173049610
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r50700037
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -233,6 +229,69 @@ private[ui] class ExecutorsPage(
</tbody>
</table>
}
+
+ private def taskData(maxTasks: Int,
+ activeTasks: Int,
--- End diff --
can you change the spacing on all the parameters to be 4 spaces indented. Put the first parameter on a newline.
Basically keep it the same as the constructor in this file private[ui] class ExecutorsPage(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172113945
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49491/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173026955
**[Test build #49727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49727/consoleFull)** for PR 10154 at commit [`2f54a33`](https://github.com/apache/spark/commit/2f54a333f8d6f9250132679a61783550de5fe34e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172128806
**[Test build #49488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49488/consoleFull)** for PR 10154 at commit [`b7a1c09`](https://github.com/apache/spark/commit/b7a1c09d483a474a023f24f8b54bafc930c34628).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173025776
Screen shot of my incoming update:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165544022
Thanks @srowen , Is there anyone we should pull into this to get more opinions on the change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174585961
**[Test build #49996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49996/consoleFull)** for PR 10154 at commit [`9293c41`](https://github.com/apache/spark/commit/9293c413d830f3c8918c5e2293b6dff5a931322d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171037858
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166499141
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r49776549
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -114,9 +114,38 @@ private[ui] class ExecutorsPage(
/** Render an HTML row representing an executor */
private def execRow(info: ExecutorSummary, logsExist: Boolean): Seq[Node] = {
+ // When GCTimePercent is edited change ToolTips.TASK_TIME to match
+ val GCTimePercent = 0.1
+
val maximumMemory = info.maxMemory
val memoryUsed = info.memoryUsed
val diskUsed = info.diskUsed
+
+ // Determine Color Opacity from 0.5-1
+ // activeTasks range from 0 to all cores
+ val activeTasksAlpha =
+ if (info.totalCores > 0) {
+ (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
--- End diff --
I didn't know about that, currently testing a change that replaces totalCores with maxTasks (= totalCores / spark.task.cpus) in my implementation. We also have the option of not shading the color on active tasks at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174656784
+1 committed this, thanks @ajbozarth
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-168765290
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-170639609
Looping in @tgravescs as the original Jira Reporter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171076004
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-165378836
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169109402
**[Test build #48776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48776/consoleFull)** for PR 10154 at commit [`34435a5`](https://github.com/apache/spark/commit/34435a5b407c09c8da6356c688878ff7a21e35c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RegressionMetrics @Since(\"2.0.0\") (`
* `trait GroupingSet extends Expression with CodegenFallback `
* `case class Cube(groupByExprs: Seq[Expression]) extends GroupingSet `
* `case class Rollup(groupByExprs: Seq[Expression]) extends GroupingSet `
* `case class Murmur3Hash(children: Seq[Expression], seed: Int) extends Expression `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166457132
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48135/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172604012
@tgravescs opinions on the changes I've made to address both you comments and the addition of the totals table?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173288603
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46854794
--- Diff: core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json ---
@@ -4,11 +4,13 @@
"rddBlocks" : 8,
"memoryUsed" : 28000128,
"diskUsed" : 0,
+ "totalCores" : 0,
--- End diff --
I updated that file according to the javadocs for HistoryServerSuite, the test that uses it. I would assume it's 0 since it's the driver and the driver has no cores assigned to run tasks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r47310707
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -33,11 +33,13 @@ private[ui] case class ExecutorSummaryInfo(
rddBlocks: Int,
memoryUsed: Long,
diskUsed: Long,
+ totalCores: Int,
--- End diff --
Did some checks, MiMa exclude is still needed even without this change and this change did not affect any code, therefore I've removed the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171811875
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171811175
**[Test build #49421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49421/consoleFull)** for PR 10154 at commit [`4294d3f`](https://github.com/apache/spark/commit/4294d3f4639be49c8031e35b41c8b59ee241b2fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-172128945
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-169149440
**[Test build #48786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48786/consoleFull)** for PR 10154 at commit [`5e27bf0`](https://github.com/apache/spark/commit/5e27bf017ac336cdaf9d27495816d97b373cdd44).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-173008898
Tested it with Yarn and it behaves the same way as Standalone
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46866604
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
+ } else {
+ ""
+ }
+ }>{info.activeTasks}</td>
+ <td style={
+ if (info.failedTasks > 0) {
+ "background:hsla(0, 100%, 50%, " + failedTasksAlpha + ");color:white"
--- End diff --
I agree, but thats just how HSLa is
http://www.w3.org/TR/css3-color/#hsla-color
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r46759351
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -128,11 +152,36 @@ private[ui] class ExecutorsPage(
<td sorttable_customkey={diskUsed.toString}>
{Utils.bytesToString(diskUsed)}
</td>
- <td>{info.activeTasks}</td>
- <td>{info.failedTasks}</td>
- <td>{info.completedTasks}</td>
+ <td style={
+ if (info.activeTasks > 0) {
+ "background:hsla(120, 100%, 25%, " + activeTasksAlpha + ");color:white"
+ } else {
+ ""
+ }
+ }>{info.activeTasks}</td>
+ <td style={
+ if (info.failedTasks > 0) {
+ "background:hsla(0, 100%, 50%, " + failedTasksAlpha + ");color:white"
--- End diff --
Another nit but if alpha is in 0.0 - 0.1 would it be clearer to write "1.0" instead of "100%" for HSL?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/10154#discussion_r49798270
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -114,9 +114,38 @@ private[ui] class ExecutorsPage(
/** Render an HTML row representing an executor */
private def execRow(info: ExecutorSummary, logsExist: Boolean): Seq[Node] = {
+ // When GCTimePercent is edited change ToolTips.TASK_TIME to match
+ val GCTimePercent = 0.1
+
val maximumMemory = info.maxMemory
val memoryUsed = info.memoryUsed
val diskUsed = info.diskUsed
+
+ // Determine Color Opacity from 0.5-1
+ // activeTasks range from 0 to all cores
+ val activeTasksAlpha =
+ if (info.totalCores > 0) {
+ (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
--- End diff --
Just updated it to look at spark.task.cpus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-174612233
**[Test build #49996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49996/consoleFull)** for PR 10154 at commit [`9293c41`](https://github.com/apache/spark/commit/9293c413d830f3c8918c5e2293b6dff5a931322d).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-166430919
I think green makes more sense for completed, and blue for active, yes. My hesitation is just that you'll end up with the completed column always sticking out since it will always end up colored. Failed too of course but those do deserve attention. Any other opinions out there? after all it's a matter mostly of taste at this point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12149] [Web UI] Executor UI improvement...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10154#issuecomment-171847217
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49428/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org