You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yaooqinn (via GitHub)" <gi...@apache.org> on 2026/04/02 07:12:56 UTC
[PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
yaooqinn opened a new pull request, #55156:
URL: https://github.com/apache/spark/pull/55156
### What changes were proposed in this pull request?
Truncate DOT graph node labels to 36 characters using middle-ellipsis when node names are long (e.g., scan nodes with fully-qualified `catalog.schema.table` paths from custom catalogs).
The full name remains available in:
- **Tooltip** (hover over the node)
- **Side panel** (click the node)
Example:
```
Before: ScanTransformer parquet spark_catalog.longschemaname.catalog_sales (60+ chars)
After: ScanTransformer pa…catalog_sales (36 chars)
```
### Why are the changes needed?
When custom catalogs generate long schema/table identifiers, the plan visualization graph becomes excessively wide and hard to read. Most operator labels are under 30 characters; only scan nodes with fully-qualified paths exceed 36 characters.
### Does this PR introduce _any_ user-facing change?
Yes. Long node labels in the SQL plan visualization are now truncated with a middle ellipsis. The full name is still accessible via tooltip and side panel.
### How was this patch tested?
- Updated existing test for truncated label in DOT output
- Added 4 new tests:
- Short labels unchanged (Sort, Filter, HashAggregate, null, empty)
- Long labels truncated with middle ellipsis (boundary, realistic scan names)
- DOT label truncated while tooltip keeps full name
- JSON side panel details keep full untruncated name
All 6 tests pass in `SparkPlanGraphSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot (Claude Opus 4.6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4181991244
<img width="951" height="1160" alt="image" src="https://github.com/user-attachments/assets/fba5d7e9-778a-47d6-8efa-ecd71d5a8452" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4178790886
### Screenshots
**Plan visualization with truncated scan node labels (query 12):**
Graph nodes show truncated labels with middle-ellipsis (`...`), keeping the operator prefix and table name:
- `Scan parquet spa...fault.fact_table` (was: `Scan parquet spark_catalog.default.fact_table`)
- `Scan parquet spa...efault.dim_table` (was: `Scan parquet spark_catalog.default.dim_table`)
**Side panel shows full untruncated name on click:**
Clicking a truncated node opens the side panel with the full name `Scan parquet spark_catalog.default.fact_table` and all metrics.
**What's NOT truncated:**
- Short operator names: `Sort`, `Filter`, `HashAggregate`, `BroadcastHashJoin`, `Project`, `Exchange` — all unchanged
- Labels without qualified paths (no `catalog.schema.table` pattern) — `Execute InsertIntoHadoopFsRelationCommand` stays full
**Truncation only activates when:**
1. Label length > 36 characters, AND
2. Label contains a dot-separated qualified name (`word.word.word` pattern)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #55156: [SPARK-56331][UI] Truncate long node labels in SQL plan visualization
URL: https://github.com/apache/spark/pull/55156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4225522979
Merged to master for Apache Spark 4.2.0. Thank you, @yaooqinn and all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4206128117
@cloud-fan we have screenshot for after, no before which you can think of a super wide WSCG cluster or a scan node
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #55156:
URL: https://github.com/apache/spark/pull/55156#discussion_r3049720842
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala:
##########
@@ -88,6 +88,27 @@ case class SparkPlanGraph(
object SparkPlanGraph {
+ /** Maximum character length for DOT graph node labels. */
+ private val MAX_LABEL_LENGTH = 36
+
+ /**
+ * Truncate a node label for compact display in the plan visualization graph.
+ * Only truncates labels containing dot-separated qualified names (e.g.,
+ * catalog.schema.table) which can become very long with custom catalogs.
+ * Uses middle-ellipsis to preserve the operator prefix and the table name.
+ * The full name is always available in the tooltip and side panel.
+ */
+ private[ui] def truncateLabel(label: String): String = {
+ if (label == null || label.length <= MAX_LABEL_LENGTH ||
+ !label.matches(".*\\w+\\.\\w+\\.\\w+.*")) {
Review Comment:
Nit: the pattern `\w+\.\w+\.\w+` could also match non-catalog dot-separated names. For example, a JDBC 3-part table name like `JDBCRelation(mydb.myschema.mytable)` on SQL Server. In practice this seems rare, and the full name is still available in the tooltip/side panel, so probably fine as-is. Just for the record.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala:
##########
@@ -88,6 +88,27 @@ case class SparkPlanGraph(
object SparkPlanGraph {
+ /** Maximum character length for DOT graph node labels. */
+ private val MAX_LABEL_LENGTH = 36
+
+ /**
+ * Truncate a node label for compact display in the plan visualization graph.
+ * Only truncates labels containing dot-separated qualified names (e.g.,
+ * catalog.schema.table) which can become very long with custom catalogs.
+ * Uses middle-ellipsis to preserve the operator prefix and the table name.
+ * The full name is always available in the tooltip and side panel.
+ */
+ private[ui] def truncateLabel(label: String): String = {
+ if (label == null || label.length <= MAX_LABEL_LENGTH ||
+ !label.matches(".*\\w+\\.\\w+\\.\\w+.*")) {
Review Comment:
`String.matches()` compiles the regex on every call. Could we precompile the pattern?
```scala
private val QUALIFIED_NAME_PATTERN = """.*\w+\.\w+\.\w+.*""".r.pattern
...
!QUALIFIED_NAME_PATTERN.matcher(label).matches()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4200634547
Thank you for pinging me, @yaooqinn
Will leave my comment tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4206010070
do we have before/after screenshots?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56331][UI] Truncate long node labels in SQL plan visualization [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55156:
URL: https://github.com/apache/spark/pull/55156#issuecomment-4223797402
Could you address https://github.com/apache/spark/pull/55156#discussion_r3049593162 , @yaooqinn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org