Re: [External Sender] Re: Bridging gap between Spark UI and Code

[Eugene Koifman <eu...@workday.com.INVALID>]

If the question is about SQL operators, Spark 3.1.3 will have Stage id displayed on SQL tab


[Diagram  Description automatically generated]

From: Wenchen Fan <cl...@gmail.com>
Date: Tuesday, May 25, 2021 at 2:31 AM
To: mhawes <ha...@gmail.com>
Cc: Spark dev list <de...@spark.apache.org>
Subject: [External Sender] Re: Bridging gap between Spark UI and Code

You can see the SQL plan node name in the DAG visualization. Please refer to https://spark.apache.org/docs/latest/web-ui.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_web-2Dui.html&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=wuPgP0XTkA0aSITo752Pl2Mh3FfB1th7k-btT_qhdIA&m=eDh5q2hvAHEG2rntK-KD9H1fXVMejRShaeOiF7d9RCs&s=SvYjj5TkJ55C5PGgmhEn0Vdfd_rwrPvyoORANgXAvJo&e=> for more details. If you still have any confusion, please let us know and we will keep improving the document.

On Tue, May 25, 2021 at 4:41 AM mhawes <ha...@gmail.com>> wrote:
@Wenchen Fan, understood that the mapping of query plan to application code
is very hard. I was wondering if we might be able to instead just handle the
mapping from the final physical plan to the stage graph. So for example
you’d be able to tell what part of the plan generated which stages. I feel
this would provide the most benefit without having to worry about several
optimisation steps.

The main issue as I see it is that currently, if there’s a failing stage,
it’s almost impossible to track down the part of the plan that generated the
stage. Would this be possible? If not, do you have any other suggestions for
this kind of debugging?

Best,
Matt



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=wuPgP0XTkA0aSITo752Pl2Mh3FfB1th7k-btT_qhdIA&m=eDh5q2hvAHEG2rntK-KD9H1fXVMejRShaeOiF7d9RCs&s=DG9A95r84k-9fq1pdS3kSi5YFGrjweVL8t92KY0VtVI&e=>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>