You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rpandya <ra...@iecommerce.com> on 2014/08/05 05:55:03 UTC
Visualizing stage & task dependency graph
Is there a way to visualize the task dependency graph of an application,
during or after its execution? The list of stages on port 4040 is useful,
but still quite limited. For example, I've found that if I don't cache() the
result of one expensive computation, it will get repeated 4 times, but it is
not easy to trace through exactly why. Ideally, what I would like for each
stage is:
- the individual tasks and their dependencies
- the various RDD operators that have been applied
- the full stack trace, both for the stage barrier, the task, and for the
lambdas used (often the RDDs are manipulated inside layers of code, so the
immediate file/line# is not enough)
Any suggestions?
Thanks,
Ravi
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Visualizing-stage-task-dependency-graph-tp11404.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Visualizing stage & task dependency graph
Posted by Phuoc Do <ph...@vida.io>.
I'm working on this patch to visualize stages:
https://github.com/apache/spark/pull/2077
Phuoc Do
On Mon, Aug 4, 2014 at 10:12 PM, Zongheng Yang <zo...@gmail.com> wrote:
> I agree that this is definitely useful.
>
> One related project I know of is Sparkling [1] (also see talk at Spark
> Summit 2014 [2]), but it'd be great (and I imagine somewhat
> challenging) to visualize the *physical execution* graph of a Spark
> job.
>
> [1] http://pr01.uml.edu/
> [2]
> http://spark-summit.org/2014/talk/sparkling-identification-of-task-skew-and-speculative-partition-of-data-for-spark-applications
>
> On Mon, Aug 4, 2014 at 8:55 PM, rpandya <ra...@iecommerce.com> wrote:
> > Is there a way to visualize the task dependency graph of an application,
> > during or after its execution? The list of stages on port 4040 is useful,
> > but still quite limited. For example, I've found that if I don't cache()
> the
> > result of one expensive computation, it will get repeated 4 times, but
> it is
> > not easy to trace through exactly why. Ideally, what I would like for
> each
> > stage is:
> > - the individual tasks and their dependencies
> > - the various RDD operators that have been applied
> > - the full stack trace, both for the stage barrier, the task, and for the
> > lambdas used (often the RDDs are manipulated inside layers of code, so
> the
> > immediate file/line# is not enough)
> >
> > Any suggestions?
> >
> > Thanks,
> >
> > Ravi
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Visualizing-stage-task-dependency-graph-tp11404.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Phuoc Do
https://vida.io/dnprock
Re: Visualizing stage & task dependency graph
Posted by Zongheng Yang <zo...@gmail.com>.
I agree that this is definitely useful.
One related project I know of is Sparkling [1] (also see talk at Spark
Summit 2014 [2]), but it'd be great (and I imagine somewhat
challenging) to visualize the *physical execution* graph of a Spark
job.
[1] http://pr01.uml.edu/
[2] http://spark-summit.org/2014/talk/sparkling-identification-of-task-skew-and-speculative-partition-of-data-for-spark-applications
On Mon, Aug 4, 2014 at 8:55 PM, rpandya <ra...@iecommerce.com> wrote:
> Is there a way to visualize the task dependency graph of an application,
> during or after its execution? The list of stages on port 4040 is useful,
> but still quite limited. For example, I've found that if I don't cache() the
> result of one expensive computation, it will get repeated 4 times, but it is
> not easy to trace through exactly why. Ideally, what I would like for each
> stage is:
> - the individual tasks and their dependencies
> - the various RDD operators that have been applied
> - the full stack trace, both for the stage barrier, the task, and for the
> lambdas used (often the RDDs are manipulated inside layers of code, so the
> immediate file/line# is not enough)
>
> Any suggestions?
>
> Thanks,
>
> Ravi
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Visualizing-stage-task-dependency-graph-tp11404.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Visualizing stage & task dependency graph
Posted by Rahul Kumar Singh <ra...@sigmoidanalytics.com>.
One way to do that is to use RDD.toDebugString to check the dependency
graph and it also gives a good idea regarding stages.
On Mon, Aug 4, 2014 at 8:55 PM, rpandya <ra...@iecommerce.com> wrote:
> Is there a way to visualize the task dependency graph of an application,
> during or after its execution? The list of stages on port 4040 is useful,
> but still quite limited. For example, I've found that if I don't cache()
> the
> result of one expensive computation, it will get repeated 4 times, but it
> is
> not easy to trace through exactly why. Ideally, what I would like for each
> stage is:
> - the individual tasks and their dependencies
> - the various RDD operators that have been applied
> - the full stack trace, both for the stage barrier, the task, and for the
> lambdas used (often the RDDs are manipulated inside layers of code, so the
> immediate file/line# is not enough)
>
> Any suggestions?
>
> Thanks,
>
> Ravi
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Visualizing-stage-task-dependency-graph-tp11404.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>