You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chenghao Lyu <ch...@cs.umass.edu> on 2023/04/07 10:22:54 UTC
[SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI
Hi,
The detailed stage page shows the involved WholeStageCodegen Ids in its DAG visualization from the Spark UI when running a SparkSQL. (e.g., under the link node:18088/history/application_1663600377480_62091/stages/stage/?id=1&attempt=0).
However, I have trouble extracting the WholeStageCodegen ids from the DAG visualization via the RESTAPIs. Is there any other way to get the WholeStageCodegen Ids information for each stage automatically?
Cheers,
Chenghao
Re: [SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
You could use QueryExecutionListener or Spark listeners to intercept query
execution events and extract whatever is required. That's what web UI does
(as it's simply a bunch of SparkListeners --> https://youtu.be/mVP9sZ6K__Y
;-)).
Pozdrawiam,
Jacek Laskowski
----
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
On Fri, Apr 7, 2023 at 12:23 PM Chenghao Lyu <ch...@cs.umass.edu> wrote:
> Hi,
>
> The detailed stage page shows the involved WholeStageCodegen Ids in its
> DAG visualization from the Spark UI when running a SparkSQL. (e.g., under
> the link
> node:18088/history/application_1663600377480_62091/stages/stage/?id=1&attempt=0).
>
> However, I have trouble extracting the WholeStageCodegen ids from the DAG
> visualization via the RESTAPIs. Is there any other way to get the
> WholeStageCodegen Ids information for each stage automatically?
>
> Cheers,
> Chenghao
>
Re: [SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI
Posted by Chitral Verma <ch...@gmail.com>.
try explain codegen on your DF and then pardee the string
On Fri, 7 Apr, 2023, 3:53 pm Chenghao Lyu, <ch...@cs.umass.edu> wrote:
> Hi,
>
> The detailed stage page shows the involved WholeStageCodegen Ids in its
> DAG visualization from the Spark UI when running a SparkSQL. (e.g., under
> the link
> node:18088/history/application_1663600377480_62091/stages/stage/?id=1&attempt=0).
>
> However, I have trouble extracting the WholeStageCodegen ids from the DAG
> visualization via the RESTAPIs. Is there any other way to get the
> WholeStageCodegen Ids information for each stage automatically?
>
> Cheers,
> Chenghao
>
Re: [SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI
Posted by Chitral Verma <ch...@gmail.com>.
try explain codegen on your DF and then pardee the string
On Fri, 7 Apr, 2023, 3:53 pm Chenghao Lyu, <ch...@cs.umass.edu> wrote:
> Hi,
>
> The detailed stage page shows the involved WholeStageCodegen Ids in its
> DAG visualization from the Spark UI when running a SparkSQL. (e.g., under
> the link
> node:18088/history/application_1663600377480_62091/stages/stage/?id=1&attempt=0).
>
> However, I have trouble extracting the WholeStageCodegen ids from the DAG
> visualization via the RESTAPIs. Is there any other way to get the
> WholeStageCodegen Ids information for each stage automatically?
>
> Cheers,
> Chenghao
>