You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by zhangliyun <ke...@126.com> on 2020/04/30 07:21:09 UTC
is there any tool to visualize the spark physical plan or spark
plan
Hi all
i want to ask a question is there any tool to visualize the spark physical plan or spark plan? sometimes the physical plan is very long so it is difficult to view it.
Best Regards
KellyZhang
Re:Re: is there any tool to visualize the spark physical plan or
spark plan
Posted by zhangliyun <ke...@126.com>.
really thanks for your suggestion
At 2020-04-30 17:30:13, "Manu Zhang" <ow...@gmail.com> wrote:
Hi Kelly,
If you can parse event log, then try listening on `SparkListenerSQLExecutionStart` event and build a `SparkPlanGraph` like https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L306.
`SparkPlanGraph` has a `makeDotFile` method where you can write out a `.dot` file and visualize it with Graphviz tools, e.g. http://www.webgraphviz.com/
Thanks,
Manu
On Thu, Apr 30, 2020 at 3:21 PM zhangliyun <ke...@126.com> wrote:
Hi all
i want to ask a question is there any tool to visualize the spark physical plan or spark plan? sometimes the physical plan is very long so it is difficult to view it.
Best Regards
KellyZhang
Re: is there any tool to visualize the spark physical plan or spark
plan
Posted by Enrico Minack <ma...@Enrico.Minack.dev>.
Kelly Zhang,
You can add a SparkListenerto your spark context:
sparkContext.addSparkListener(newSparkListener{})
That one can override onTaskEnd, which provides you a
SparkListenerTaskEnd for each task. That instance provides you access to
the metrics.
See:
-
https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListener.html#onTaskEnd-org.apache.spark.scheduler.SparkListenerTaskEnd-
-
https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListenerTaskEnd.html#taskMetrics--
Regards,
Enrico
Am 02.05.20 um 00:45 schrieb zhangliyun:
>
>
>
> Hi Wenchen Fan:
> thanks for reply. in the link, i saw sql metrics which is very userful.
> ```
> SQL metrics
>
> The metrics of SQL operators are shown in the block of physical
> operators. The SQL metrics can be useful when we want to dive into the
> execution details of each operator. For example, “number of output
> rows” can answer how many rows are output after a Filter operator,
> “shuffle bytes written total” in an Exchange operator shows the number
> of bytes written by a shuffle.
>
> Here is the list of SQL metrics:
>
> ````
>
> my question is except reading these metrics in the spark web ui., is
> there any way to read the metrics in driver side by code?
>
>
> Best regards
>
> Kelly Zhang
>
>
>
> At 2020-04-30 21:38:56, "Wenchen Fan" <cl...@gmail.com> wrote:
>
> Does the Spark SQL web UI work for you?
> https://spark.apache.org/docs/3.0.0-preview/web-ui.html#sql-tab
>
> On Thu, Apr 30, 2020 at 5:30 PM Manu Zhang
> <owenzhang1990@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Kelly,
>
> If you can parse event log, then try listening on
> `SparkListenerSQLExecutionStart` event and build a
> `SparkPlanGraph` like
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L306.
>
> `SparkPlanGraph` has a `makeDotFile` method where you can
> write out a `.dot` file and visualize it with Graphviz tools,
> e.g. http://www.webgraphviz.com/
>
> Thanks,
> Manu
>
> On Thu, Apr 30, 2020 at 3:21 PM zhangliyun <kellyzly@126.com
> <ma...@126.com>> wrote:
>
> Hi all
> i want to ask a question is there any tool to visualize
> the spark physical plan or spark plan? sometimes the
> physical plan is very long so it is difficult to view it.
>
> Best Regards
> KellyZhang
>
>
>
>
Re:Re: is there any tool to visualize the spark physical plan or
spark plan
Posted by zhangliyun <ke...@126.com>.
Hi Wenchen Fan:
thanks for reply. in the link, i saw sql metrics which is very userful.
```
SQL metrics
The metrics of SQL operators are shown in the block of physical operators. The SQL metrics can be useful when we want to dive into the execution details of each operator. For example, “number of output rows” can answer how many rows are output after a Filter operator, “shuffle bytes written total” in an Exchange operator shows the number of bytes written by a shuffle.
Here is the list of SQL metrics:
````
my question is except reading these metrics in the spark web ui., is there any way to read the metrics in driver side by code?
Best regards
Kelly Zhang
At 2020-04-30 21:38:56, "Wenchen Fan" <cl...@gmail.com> wrote:
Does the Spark SQL web UI work for you?
https://spark.apache.org/docs/3.0.0-preview/web-ui.html#sql-tab
On Thu, Apr 30, 2020 at 5:30 PM Manu Zhang <ow...@gmail.com> wrote:
Hi Kelly,
If you can parse event log, then try listening on `SparkListenerSQLExecutionStart` event and build a `SparkPlanGraph` like https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L306.
`SparkPlanGraph` has a `makeDotFile` method where you can write out a `.dot` file and visualize it with Graphviz tools, e.g. http://www.webgraphviz.com/
Thanks,
Manu
On Thu, Apr 30, 2020 at 3:21 PM zhangliyun <ke...@126.com> wrote:
Hi all
i want to ask a question is there any tool to visualize the spark physical plan or spark plan? sometimes the physical plan is very long so it is difficult to view it.
Best Regards
KellyZhang
Re: is there any tool to visualize the spark physical plan or spark plan
Posted by Wenchen Fan <cl...@gmail.com>.
Does the Spark SQL web UI work for you?
https://spark.apache.org/docs/3.0.0-preview/web-ui.html#sql-tab
On Thu, Apr 30, 2020 at 5:30 PM Manu Zhang <ow...@gmail.com> wrote:
> Hi Kelly,
>
> If you can parse event log, then try listening on
> `SparkListenerSQLExecutionStart` event and build a `SparkPlanGraph` like
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L306
> .
>
> `SparkPlanGraph` has a `makeDotFile` method where you can write out a
> `.dot` file and visualize it with Graphviz tools, e.g.
> http://www.webgraphviz.com/
>
> Thanks,
> Manu
>
> On Thu, Apr 30, 2020 at 3:21 PM zhangliyun <ke...@126.com> wrote:
>
>> Hi all
>> i want to ask a question is there any tool to visualize the spark
>> physical plan or spark plan? sometimes the physical plan is very long so it
>> is difficult to view it.
>>
>> Best Regards
>> KellyZhang
>>
>>
>>
>>
>
Re: is there any tool to visualize the spark physical plan or spark plan
Posted by Manu Zhang <ow...@gmail.com>.
Hi Kelly,
If you can parse event log, then try listening on
`SparkListenerSQLExecutionStart` event and build a `SparkPlanGraph` like
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L306
.
`SparkPlanGraph` has a `makeDotFile` method where you can write out a
`.dot` file and visualize it with Graphviz tools, e.g.
http://www.webgraphviz.com/
Thanks,
Manu
On Thu, Apr 30, 2020 at 3:21 PM zhangliyun <ke...@126.com> wrote:
> Hi all
> i want to ask a question is there any tool to visualize the spark
> physical plan or spark plan? sometimes the physical plan is very long so it
> is difficult to view it.
>
> Best Regards
> KellyZhang
>
>
>
>