You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "khalidmammadov (via GitHub)" <gi...@apache.org> on 2023/11/03 19:41:51 UTC
[PR] [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString() [spark]
khalidmammadov opened a new pull request, #43651:
URL: https://github.com/apache/spark/pull/43651
### What changes were proposed in this pull request?
This PR adds a new method df.explainString() which produces the same output as df.explain() but instead of printing it returns the output as String. This output then can be manipupated easily by users and saved to external drives.
This is frequently needed feature for performance optimization purposes. Users often want to look into this output in running systems and so would like to save/extract this output from running systems (with some settings toggle) for later analysis.
Current API only provided for Scala i.e.
`df.queryExecution.toString()`
and also not located in intuitive place where average Spark user (i.e. non Expert/Scala dev) can see it immediately.
For Python there are various workaround provided in Stackoverflow pages most of which suggesting to use internal Spark APIs e.g.:
https://stackoverflow.com/questions/54124386/capturing-the-result-of-explain-in-pyspark
Alternatively, I see most ofthen Python users just capturing std output as below which is another workaround and not first choice for regular Spark/Pyhton users.
```
with io.StringIO() as buf ...`:
df.explain(True)
```
So, it would help users a lot to have this output avalilable as:
`df.explainString()`
i.e. next to
`df.explain()`
so users can easily locate it and use.
### Why are the changes needed?
To help users debug performance issues on running systems
### Does this PR introduce _any_ user-facing change?
Yes, adds a new DataFrame method
### How was this patch tested?
New unit tests
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString() [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43651:
URL: https://github.com/apache/spark/pull/43651#discussion_r1382306692
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -585,16 +585,10 @@ class Dataset[T] private[sql](
* @group basic
* @since 3.0.0
*/
- def explain(mode: String): Unit = sparkSession.withActive {
- // Because temporary views are resolved during analysis when we create a Dataset, and
- // `ExplainCommand` analyzes input query plan and resolves temporary views again. Using
- // `ExplainCommand` here will probably output different query plans, compared to the results
- // of evaluation of the Dataset. So just output QueryExecution's query plans here.
-
+ def explain(mode: String): Unit =
Review Comment:
Hmmm .. can't you just hijack stdout and save it into a string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString() [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43651:
URL: https://github.com/apache/spark/pull/43651#issuecomment-1939855929
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString() [spark]
Posted by "khalidmammadov (via GitHub)" <gi...@apache.org>.
khalidmammadov commented on code in PR #43651:
URL: https://github.com/apache/spark/pull/43651#discussion_r1382330036
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -585,16 +585,10 @@ class Dataset[T] private[sql](
* @group basic
* @since 3.0.0
*/
- def explain(mode: String): Unit = sparkSession.withActive {
- // Because temporary views are resolved during analysis when we create a Dataset, and
- // `ExplainCommand` analyzes input query plan and resolves temporary views again. Using
- // `ExplainCommand` here will probably output different query plans, compared to the results
- // of evaluation of the Dataset. So just output QueryExecution's query plans here.
-
+ def explain(mode: String): Unit =
Review Comment:
I can and do and see others also do, so was thinking looks like missing feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString() [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #43651: [SPARK-45782][CORE][PYTHON] Add Dataframe API df.explainString()
URL: https://github.com/apache/spark/pull/43651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org