You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "j1wonpark (via GitHub)" <gi...@apache.org> on 2026/06/22 22:59:50 UTC
[PR] [SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server [spark]
j1wonpark opened a new pull request, #56682:
URL: https://github.com/apache/spark/pull/56682
### What changes were proposed in this pull request?
This PR makes the Spark Connect UI tab render in the Spark History Server (SHS). Two changes:
1. Register `SparkConnectServerHistoryServerPlugin` via the `AppHistoryServerPlugin` SPI (a new `META-INF/services/org.apache.spark.status.AppHistoryServerPlugin` resource in the `connect/server` module), so that SHS discovers it through `ServiceLoader`, as the SQL, Streaming and Hive Thrift Server plugins already do.
2. Make `SparkConnectServerListener` read its configuration from the `SparkConf` passed to its constructor instead of `SparkEnv.get.conf`.
### Why are the changes needed?
The Spark Connect UI page added in SPARK-44394 works on a live driver but never appears in the History Server:
- `SparkConnectServerHistoryServerPlugin` is implemented but is not registered in `META-INF/services`, so SHS never loads it via `ServiceLoader`. The live UI works only because `SparkConnectService` registers the tab and listener directly.
- Even once the plugin is registered, the listener's constructor reads configuration via `SparkEnv.get.conf`. There is no active `SparkEnv` during SHS replay, so `SparkEnv.get` returns `null` and the listener throws an NPE in `FsHistoryProvider.rebuildAppStore`, failing the whole application UI with HTTP 500:
```
java.lang.NullPointerException: Cannot invoke "org.apache.spark.SparkEnv.conf()" because the return value of "org.apache.spark.SparkEnv$.get()" is null
at org.apache.spark.sql.connect.ui.SparkConnectServerListener.<init>(SparkConnectServerListener.scala)
at org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin.createListeners(SparkConnectServerHistoryServerPlugin.scala)
at org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala)
```
### Does this PR introduce _any_ user-facing change?
Yes. The Spark Connect tab (sessions / executions) now renders in the History Server when replaying event logs of a Spark Connect application. Previously the tab was never shown there. This is a change relative to released versions and master.
### How was this patch tested?
- Added a unit test in `SparkConnectServerListenerSuite` that constructs the listener with no active `SparkEnv` (simulating History Server replay). It reproduces the NPE before the fix and passes after.
- Updated the existing test helper to set the UI-retention configs on the `SparkConf` passed to the listener (previously set on `SparkEnv.get.conf`, which the listener no longer reads).
- Manually verified on a History Server that the Connect tab renders from Spark Connect event logs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #56682: [SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server
URL: https://github.com/apache/spark/pull/56682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #56682:
URL: https://github.com/apache/spark/pull/56682#issuecomment-4780635504
thanks, merged to master and backported to branch-4.0 onwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #56682:
URL: https://github.com/apache/spark/pull/56682#issuecomment-4780432868
Manually verified that Connect tab correctly is rendered in SHS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org