You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dnprock <gi...@git.apache.org> on 2014/08/21 07:15:34 UTC
[GitHub] spark pull request: [SPARK-975] Visual Debugger of stages and call...
GitHub user dnprock opened a pull request:
https://github.com/apache/spark/pull/2077
[SPARK-975] Visual Debugger of stages and callstacks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dnprock/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2077.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2077
----
commit df94cc02e9b5cdddf46f312ad82d4cc6a4ea4c89
Author: Phuoc Do <dn...@gmail.com>
Date: 2014-08-21T05:13:50Z
Visual Ddebugger of stages and callstack
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53230539
Any update on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53949461
Yes definitely need to wait for that in order to review this one. Without the job page, there is no place to actually put this visualization. Thanks for understanding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53367931
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53513626
Hey @dnprock,
I played with this a little bit on my laptop. While I like the idea to visualize the dag, I think the current implementation is not ideal for the following reasons:
1. It shows a single job visualization in the stage page, even in the case of multiple jobs.
2. Related to 1, we probably want to add a job level page to the web ui, and then put something similar to this visualization there.
3. The arrangement of space is not ideal (in circular shape). Ideally we would want something similar to the usual ways people render trees.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-55440393
The "didn't merge cleanly" here is a false-positive due to a bug in the pull request builder; please ignore the spurious warning (I'll fix this in the afternoon).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975] Visual debugger of stages and call...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-52891108
Would you mind to add `[Core]` after `[SPARK-975]` in the PR title?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by wbraik <gi...@git.apache.org>.
Github user wbraik commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-89632207
Does anyone have a good example of an application which produces multiple (different) jobs, that we could use to test this on ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53652475
A single job can trigger multiple stages. A job becomes a tree of stages essentially.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53510433
Nope, you shouldn't add a license to d3.js and .css. Please remove that -- we can't change their copyright. Instead, add them to .rat-excludes so they don't get automatically checked.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53509769
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975] Visual Debugger of stages and call...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-52879538
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-55440242
QA tests have started for PR 2077. This patch DID NOT merge cleanly! <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/78/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53510501
In addition we need to include D3's license in our LICENSE file. Take a look at how other libraries are listed there (e.g. sorttable.js).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53904279
@rxin do we wait until Job to merge this visualization?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-96770251
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53474433
@rxin @mateiz Looks like I need to add Apache license. d3.js uses BSD. Do I add Apache on top of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975] Visual debugger of stages and call...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-52887465
Can you upload a screenshot of the UI?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53654943
I'd be interesting to see what you come up with for rendering DAGs with tree layouts. I faced a similar problem last year when working on a visualization class project called [DeepViz](https://github.com/bruckner/deepViz). In our project, we wanted to display a DAG that described the layers of a multi-layer neutral network while supporting interaction with the DAG (e.g. click on a layer to change the main display).
We ended up using GraphViz to render a SVG and used JQuery to attach event handlers to the rendered SVG elements. I think that there may be libraries that use the output of GraphViz's layout engine (running on the server side) but render the actual graph in D3.
I recently used a Graphviz + SVG + Javascript setup in a research prototype (we were visualizing large program lineage graphs); I chose this solution instead of the pure-Javascript `.dot`-to-SVG libraries because our graphs were fairly complicated and those libraries didn't support many features of the DOT language.
My guess is that we won't want to pull in GraphViz as a dependency, though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53654914
Stages within a job can also form a DAG instead of a tree, e.g.:
```scala
val a = sc.textFile("file").map(???).reduceByKey(???)
val b = a.filter(???).reduceByKey(???)
val c = a.map(???).reduceByKey(???)
val d = (b ++ c).reduceByKey(???)
d.collect()
```
Each RDD introduces a shuffle in the case above, and the stage DAG looks roughly like:
```
ar,bm (The DAG grows from left to right)
/ \
am br,cr,dm - dr
\ /
ar,cm
```
where `xm` and `xr` means the mapper and reducer side of the shuffle introduced by RDD `x` respectively.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by velvia <gi...@git.apache.org>.
Github user velvia commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53846867
Ah yes, the job group.
On Fri, Aug 29, 2014 at 12:47 AM, Reynold Xin <no...@github.com>
wrote:
> Couple different alternatives:
>
> 1.
>
> Just provide a job to stage mapping for now, since that's a huge step
> up from what we have right now which is stage only.
> 2.
>
> There is this experimental thing called a job group ...
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/2077#issuecomment-53846679>.
>
--
The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53368278
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19191/consoleFull) for PR 2077 at commit [`df94cc0`](https://github.com/apache/spark/commit/df94cc02e9b5cdddf46f312ad82d4cc6a4ea4c89).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` $FWDIR/bin/spark-submit --class org.apache.spark.repl.Main "$`
* ` $FWDIR/bin/spark-submit --class org.apache.spark.repl.Main "$`
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable `
* `class BoundedFloat(float):`
* `class JoinedRow2 extends Row `
* `class JoinedRow3 extends Row `
* `class JoinedRow4 extends Row `
* `class JoinedRow5 extends Row `
* `class GenericRow(protected[sql] val values: Array[Any]) extends Row `
* `abstract class MutableValue extends Serializable `
* `final class MutableInt extends MutableValue `
* `final class MutableFloat extends MutableValue `
* `final class MutableBoolean extends MutableValue `
* `final class MutableDouble extends MutableValue `
* `final class MutableShort extends MutableValue `
* `final class MutableLong extends MutableValue `
* `final class MutableByte extends MutableValue `
* `final class MutableAny extends MutableValue `
* `final class SpecificMutableRow(val values: Array[MutableValue]) extends MutableRow `
* `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate `
* `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression `
* `case class CollectHashSetFunction(`
* `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression `
* `case class CombineSetsAndCountFunction(`
* `case class CountDistinctFunction(`
* `case class MaxOf(left: Expression, right: Expression) extends Expression `
* `case class NewSet(elementType: DataType) extends LeafExpression `
* `case class AddItemToSet(item: Expression, set: Expression) extends Expression `
* `case class CombineSets(left: Expression, right: Expression) extends BinaryExpression `
* `case class CountSet(child: Expression) extends UnaryExpression `
* `case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975] [core] Visual debugger of stages a...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-52934791
Attached a screenshot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53660819
@rxin @liancheng @JoshRosen, I use dagre-d3 package to generate DAG. It looks like below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-61545640
Any update on this patch merge? I can resync my branch. @liancheng @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53846679
Couple different alternatives:
1. Just provide a job to stage mapping for now, since that's a huge step up from what we have right now which is stage only.
2. There is this experimental thing called a job group ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-54694428
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-54006224
@rxin let me know when job page is in. If there are more visualization tasks, let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-55449348
QA results for PR 2077:<br>- This patch PASSES unit tests.<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/78/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53652400
@rxin force layout is the only layout that works with DAG in D3. I'm experimenting with some custom layout for tree structure.
How are jobs and stages related?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by dnprock <gi...@git.apache.org>.
Github user dnprock commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53475389
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53368225
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19191/consoleFull) for PR 2077 at commit [`df94cc0`](https://github.com/apache/spark/commit/df94cc02e9b5cdddf46f312ad82d4cc6a4ea4c89).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-975][CORE] Visual debugger of stages an...
Posted by velvia <gi...@git.apache.org>.
Github user velvia commented on the pull request:
https://github.com/apache/spark/pull/2077#issuecomment-53846528
@rxin what characteristic of each stage allows you to group relevant stages together into "jobs"? For example the job server itself can run multiple "jobs" on a single context, and it knows how to group things together, but how would Spark itself know?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org