You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "alexandrefimov (via GitHub)" <gi...@apache.org> on 2026/05/30 16:08:48 UTC
[PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
alexandrefimov opened a new pull request, #56225:
URL: https://github.com/apache/spark/pull/56225
### What changes were proposed in this pull request?
This PR preserves the original SQL physical plan inside `SQLExecutionRDD` and uses it when building `SparkPlanInfo` for `RDDScanExec`.
When a DataFrame is converted to an RDD and then back to a DataFrame, Spark currently shows only `Scan ExistingRDD` in the SQL UI. This change lets `SparkPlanInfo` discover the underlying SQL plan from the `SQLExecutionRDD` lineage and attach it under the `RDDScanExec` node.
The RDD lineage traversal is defensive:
- it deduplicates RDDs by id;
- it stops at each discovered `SQLExecutionRDD`;
- it tolerates dependency lookup failures by skipping that branch.
### Why are the changes needed?
Without this, the SQL UI loses useful operator visibility and metrics after workflows such as:
```scala
val source = spark.range(10).filter($"id" > 3)
val recreated = spark.createDataFrame(source.rdd, source.schema)
recreated.collect()
```
The UI only exposes `Scan ExistingRDD`, while the internal SQL operators, such as `Filter`, are hidden. This makes query diagnosis harder for DataFrame/RDD/DataFrame workflows.
### Does this PR introduce _any_ user-facing change?
Yes. For SQL executions that scan an RDD backed by a previous SQL execution, the SQL UI can now show the internal SQL plan below `Scan ExistingRDD`, including metrics such as `number of output rows`.
### How was this patch tested?
Added regression coverage in `SparkPlanInfoSuite` for:
- exposing the internal SQL plan under `RDDScanExec`;
- preserving SQL metrics in the SQL UI plan graph;
- deduplicating shared RDD lineage;
- nested `createDataFrame(df.rdd, df.schema)` cases.
Ran:
```bash
build/sbt "sql/testOnly *SparkPlanInfoSuite -- -z SPARK-47017"
```
Result:
```text
Tests: succeeded 1, failed 0
All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4908275102
Rebased onto the latest master.
I also ran the focused local check with JDK 17:
build/sbt "sql/testOnly org.apache.spark.sql.execution.ui.SparkPlanInfoSuite"
It passed locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-5169994283
Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4587325469
Hi @holdenk_amp, this is ready for review when you have a chance :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4992732095
Hi @holdenk, the fork CI rerun on the rebased head completed successfully, including the two jobs that previously failed during disk cleanup. GitHub still shows the original failed Build check on the upstream PR, but the rerun itself is green. When you have time, could you take another look and advise whether this should target branch-4?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-5168943254
Sorry this looks good, merged to branch-4.x and master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4639944124
Hi @holdenk, just a gentle follow-up on this PR when you have a chance. CI is green and the change is fairly small. If there’s someone else I should ask for review, please let me know. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4908121004
Ah my bad I've been distracted with other tasks. Lets rebase it and then yeah we can look at if this fits into branch-4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4908108696
Hi @holdenk, just checking in on this PR when you have a chance.
The PR is still mergeable, but the last CI run is from May 30 and master has moved quite a bit since then. I’m happy to rebase and refresh CI if that would help with review.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-5166882964
@holdenk, gentle follow-up when you have a chance. The rebased head fork CI rerun is green. Could you advise whether this should target branch-4 and take another look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #56225: [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec
URL: https://github.com/apache/spark/pull/56225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47017][SQL] Show internal SQL plans for RDDScanExec [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #56225:
URL: https://github.com/apache/spark/pull/56225#issuecomment-4640042436
I like this change let me try and review it on Monday. I know there's another committer who's been doing UI work I'll loop in for a second pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org