[jira] [Updated] (SPARK-57601) Spark Connect UI tab is unavailable in the Spark History Server

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-57601:
-----------------------------------
    Labels: pull-request-available  (was: )

> Spark Connect UI tab is unavailable in the Spark History Server
> ---------------------------------------------------------------
>
>                 Key: SPARK-57601
>                 URL: https://issues.apache.org/jira/browse/SPARK-57601
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, Web UI
>    Affects Versions: 4.1.2
>            Reporter: Jiwon Park
>            Priority: Major
>              Labels: pull-request-available
>
> The Spark Connect UI page (SPARK-44394) works on a live driver but never appears in the Spark History Server (SHS). Two separate problems are involved.
> h3. 1. The plugin is not registered for the History Server
> {{SparkConnectServerHistoryServerPlugin}} implements {{AppHistoryServerPlugin}}, but it is not declared in {{META-INF/services/org.apache.spark.status.AppHistoryServerPlugin}} (the SQL, Streaming and Hive Thrift Server plugins are). As a result the History Server never discovers it via {{ServiceLoader}}, even though the class is on the classpath. The live UI works only because {{SparkConnectService}} registers the tab and listener directly.
> h3. 2. The listener throws NPE during History Server replay
> Once the plugin is registered, {{FsHistoryProvider.rebuildAppStore}} calls {{AppHistoryServerPlugin.createListeners}} -> {{new SparkConnectServerListener}}, whose constructor reads configuration via {{SparkEnv.get.conf}}. There is no active {{SparkEnv}} in the History Server ({{SparkEnv.get}} returns null), so it throws an NPE and the whole application UI rebuild fails with HTTP 500:
> {code}
> java.lang.NullPointerException: Cannot invoke "org.apache.spark.SparkEnv.conf()" because the return value of "org.apache.spark.SparkEnv$.get()" is null
>   at org.apache.spark.sql.connect.ui.SparkConnectServerListener.<init>(SparkConnectServerListener.scala:49)
>   at org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin.createListeners(SparkConnectServerHistoryServerPlugin.scala:30)
>   at org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:1189)
> {code}
> The listener already receives a {{SparkConf}} through its constructor, so it should read from that instead of {{SparkEnv.get.conf}}.
> h3. How to reproduce
> Enable {{spark.eventLog.enabled}}, run a Spark Connect session so Connect events are written to the event log, then open that application in the History Server. Every page returns HTTP 500.
> h3. Proposed fix
> * Register {{SparkConnectServerHistoryServerPlugin}} via the {{AppHistoryServerPlugin}} SPI.
> * Use the constructor's {{sparkConf}} instead of {{SparkEnv.get.conf}} in {{SparkConnectServerListener}}.
> Both are needed for the Spark Connect tab to render from event logs in the History Server. This affects all 4.x releases (and master).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org