You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shroffpradyumn <gi...@git.apache.org> on 2015/04/17 02:00:17 UTC
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
GitHub user shroffpradyumn opened a pull request:
https://github.com/apache/spark/pull/5547
[SPARK-6418] Add simple per-stage visualization to the UI [WIP]
I'm working on adding a graph-visualization to the Spark jobs page to analyze the data visually. This is a working version, and it would be great if I could get some feedback on what changes you'd like to see.
NOTE: This is a work in progress, and I'm currently working on adding the following changes:
1. Changing the unit of the x-axis (from ms to s to minutes) depending on the time taken by the longest job.
2. Reducing the tick count for the y-axis when the total # of tasks is ~50 or more.
3. Eliminate the shading for the launch time (keeping it transparent) as it can potentially confuse users.
4. Breaking down the duration phase into shuffle read, write time, etc.
Any feedback is greatly appreciated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shroffpradyumn/spark graph3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5547.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5547
----
commit a2141c484c51f068eb1f5c67d3f8984dd49633e0
Author: Pradyumn Shroff <sh...@airbears2-10-142-132-205.airbears2.1918.berkeley.edu>
Date: 2015-04-14T06:34:20Z
basic framework with only scheduler delays
commit a6f92e2711c8f654229ecae58d9a01df032cc4b5
Author: Pradyumn Shroff <sh...@pradyumnmac.local>
Date: 2015-04-15T05:12:43Z
working version
commit 1035f121173d9f8cd888f4e0064184cba4074cd0
Author: Pradyumn Shroff <sh...@pradyumnmac.local>
Date: 2015-04-15T06:38:06Z
good looking version (without hover)
commit a4d52079a510e02e029cdd284e305a4ab1f10ef4
Author: Pradyumn Shroff <sh...@pradyumnmac.local>
Date: 2015-04-15T09:07:34Z
final version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by shroffpradyumn <gi...@git.apache.org>.
Github user shroffpradyumn commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95797940
Thank you all for your feedback, and I apologize for my late reply (it’s been a rough week of midterms).
@pwendell - I’ve addressed all your inline comments (memoization, Javascript indentation, JSON lists, etc.) in my latest commit. As per the load time of the graph, it’s improved a bit after moving from string representations to JSON arrays, but only by a small factor.
When you say you’re skeptical about the graph scalability, what is the maximum number of tasks you want displayed on the graph? I’m thinking of keeping it to 1000 (at the most), and having the users select a task range if they want to view a different region of tasks (say tasks 1200-2000 for example).
My reason for the above is that the task stages become too cluttered above a certain number, so it’s better to keep a limit, or alternatively, increase the max height of the graph (which would involve a lot more scrolling though).
@andrewor14 - The visualization doesn’t currently support zooming, and it will definitely be pretty challenging to implement it on top of D3.js. However, the task-range functionality I mentioned above can serve as a pseudo-zoom feature since a user can select a task range and hence zoom into the graph.
Also, breaking down the task times along the vertical axis shouldn’t be that difficult so we can definitely add that later on if required (provided this patch gets accepted haha).
@punya - I haven’t looked into using Amber yet, and I’ll definitely check out plottable.js.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/5547#discussion_r28743686
--- Diff: core/src/main/resources/org/apache/spark/ui/static/jobs-graph.js ---
@@ -0,0 +1,118 @@
+function renderJobsGraphs(data) {
+ /* show visualization toggle */
+ $(".expand-visualization-arrow").toggleClass('arrow-closed');
+ $(".expand-visualization-arrow").toggleClass('arrow-open');
+ if ($(".expand-visualization-arrow").hasClass("arrow-closed")) {
+ $("#chartContainer").empty();
+ return;
+ }
+
+ /* no data to graph */
+ if (!Object.keys(data).length) {
+ return;
+ }
+
+ /* format data to a form readable by dimple.js */
+ var tableData = [];
+ for (var k in data) {
+ var arr = (data[k]).split(",");
--- End diff --
If you pass each of the inputs as lists, I think you won't need to do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-93865048
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94585789
[Test build #30610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30610/consoleFull) for PR 5547 at commit [`7fac1eb`](https://github.com/apache/spark/commit/7fac1eb96c61cb23e020aa55c306f1b698e4196b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-96840735
If there are over 1000 tasks, it seems like it would be more valuable to see
* statistical information about all the tasks
* details about a smaller subset chosen in some way (like a drill down view)
Rendering 10k tasks is possible using a canvas, but it's unclear to me what
a user would do with that much density of information.
Punya
On Mon, Apr 27, 2015 at 6:33 PM Pradyumn Shroff <no...@github.com>
wrote:
> After some further research, I have come to the conclusion that if we want
> to have over a 1000 tasks displayed on the graph, d3.js or libraries on top
> of it aren't the best choice since rendering that many SVG elements is
> bound to be slow above a certain limit.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5547#issuecomment-96839375>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/5547#discussion_r28743323
--- Diff: core/src/main/resources/org/apache/spark/ui/static/jobs-graph.js ---
@@ -0,0 +1,118 @@
+function renderJobsGraphs(data) {
+ /* show visualization toggle */
--- End diff --
Can you use 2 spaces for indents throughout all javascript files, instead of tap characters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94585111
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95420852
Also, if you're looking for a reasonable path to panning/zooming, you might want to take a look at http://plottablejs.org/.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-93864988
cc @andrewor14 @pwendell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94607373
Thanks a lot for submitting this. It is a cool feature - we'll need to think about whether we like this charting library vs the one in the timeline view PR. I am going to defer to @kayousterhout to give a more thorough review, but I mentioned a few things inline.
I am a bit concerned about the scalability here. I tried a job locally with 1000 tasks and it took more than 10 seconds to generate the graph. It would be good to explore what part takes a long time. I did some quick profiling and it looks like getOffsetHeight in the dimple library was the culprit... that may be tough to improve on.
Also, it might be nice to memoize the rendered graph in case someone opens and closes the tab multiple times.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by shroffpradyumn <gi...@git.apache.org>.
Github user shroffpradyumn commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-96839375
After some further research, I have come to the conclusion that if we want to have over a 1000 tasks displayed on the graph, d3.js or libraries on top of it aren't the best choice since rendering that many SVG elements is bound to be slow above a certain limit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-96767076
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95223629
Given the increasing complexity of the status pages' UI logic, does it make sense to move from manually toggling CSS classes using jQuery, to a modern single page application framework such as Angular, React or Ember?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-93865113
Jenkins, this is ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95789429
[Test build #30910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30910/consoleFull) for PR 5547 at commit [`5c3a2a6`](https://github.com/apache/spark/commit/5c3a2a697fca83d6de843850e786cf3406c4bd5a).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/5547#discussion_r28743664
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -234,6 +235,8 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
val deserializationTimes = validTasks.map { case TaskUIData(_, metrics, _) =>
metrics.get.executorDeserializeTime.toDouble
}
+ graphData("Task Deserialization Time") = deserializationTimes.mkString(",")
--- End diff --
Rather than creating your own string representation of a list, can do the necessary conversation to pass proper JSON lists here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94585961
@pwendell talked about this and #2342 a little bit offline. Our feeling is that this is a more elegant representation of task times than #2342, especially when there are many tasks within a stage. One concern I have, however, what happens when you zoom (does it currently support zooming?). It would make little sense to zoom without keeping the axes, but my impression is that implementing this is pretty hard since we're directly using d3.
Bonus: It doesn't have to be part of this patch, but it would really cool if there's a mode where we can align the breakdown of the task times along the vertical axis. Right now you can't really compare the serialization time of the first task with that of the last task, let alone track whether it has grown incrementally over time. Realistically we will implement this separately say for 1.5, but I imagine this bonus feature is gonna be immensely useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94585898
[Test build #30610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30610/consoleFull) for PR 5547 at commit [`7fac1eb`](https://github.com/apache/spark/commit/7fac1eb96c61cb23e020aa55c306f1b698e4196b).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-94585899
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30610/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95789315
[Test build #30910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30910/consoleFull) for PR 5547 at commit [`5c3a2a6`](https://github.com/apache/spark/commit/5c3a2a697fca83d6de843850e786cf3406c4bd5a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by shroffpradyumn <gi...@git.apache.org>.
Github user shroffpradyumn commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-97274549
The Spark administrators have to decided to go forward with #2342 so I'm closing this pull-request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/5547#discussion_r28744020
--- Diff: core/src/main/resources/org/apache/spark/ui/static/jobs-graph.js ---
@@ -0,0 +1,118 @@
+function renderJobsGraphs(data) {
+ /* show visualization toggle */
+ $(".expand-visualization-arrow").toggleClass('arrow-closed');
+ $(".expand-visualization-arrow").toggleClass('arrow-open');
+ if ($(".expand-visualization-arrow").hasClass("arrow-closed")) {
+ $("#chartContainer").empty();
+ return;
+ }
+
+ /* no data to graph */
+ if (!Object.keys(data).length) {
+ return;
+ }
+
+ /* format data to a form readable by dimple.js */
+ var tableData = [];
+ for (var k in data) {
+ var arr = (data[k]).split(",");
+ data[k] = arr;
+ }
+ var startTime = getMin(data["launchtime"]);
+ var numTasks = Math.min(1000, data[k].length);
+
+ /*data update */
+ data["launchtime"] = data["launchtime"].map(function (launchTime) {return launchTime-startTime;});
+ var maxTime = 0;
+ for (i = 0; i < numTasks; i++) {
+ var time = 0;
+ for (var key in data) {
+ time += parseFloat(data[key][i]);
--- End diff --
this might be pretty slow when there are thousands of tasks - if so, sending proper double types in JSON would be faster.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by shroffpradyumn <gi...@git.apache.org>.
Github user shroffpradyumn commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-96842551
That's exactly my viewpoint but to be very honest, I haven't used Spark much so I'm not sure what the average use case is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5547#issuecomment-95789435
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30910/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6418] Add simple per-stage visualizatio...
Posted by shroffpradyumn <gi...@git.apache.org>.
Github user shroffpradyumn closed the pull request at:
https://github.com/apache/spark/pull/5547
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org