You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/08/04 09:00:54 UTC
[PR] [CONNECT] Show ML cache status in Spark Connect UI [spark]
zhengruifeng opened a new pull request, #57754:
URL: https://github.com/apache/spark/pull/57754
### What changes were proposed in this pull request?
This PR adds a live ML cache status column to the session table in the Spark Connect UI.
For active sessions, the column shows the number of objects currently in memory and the current and maximum in-memory and total cache sizes. It also reports when memory control is disabled or the ML cache has not been used. Closed sessions and History Server views show `N/A` because their live cache state is unavailable.
The UI reads the cache status without updating the session's last-access time or initializing an unused ML cache.
### Why are the changes needed?
Spark Connect ML models are cached per session, but operators currently cannot inspect cache utilization in the Spark UI. Showing the live status makes it easier to understand model retention, memory pressure, and disk offloading for each session.
### Does this PR introduce _any_ user-facing change?
Yes. The Spark Connect session table has a new `ML Cache` column showing live per-session cache utilization.
### How was this patch tested?
Added unit coverage for ML cache status collection and Connect UI rendering.
Tests have not been run yet; this is a draft PR. `git diff --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #57754:
URL: https://github.com/apache/spark/pull/57754#discussion_r3757953371
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:
##########
@@ -431,8 +521,152 @@ private[ui] class SessionStatsPagedTable(
<td> {if (session.finishTimestamp > 0) formatDate(session.finishTimestamp)} </td>
<td> {formatDurationVerbose(session.totalTime)} </td>
<td> {session.totalExecution.toString} </td>
+ <td> {renderMLCacheStatus(session)} </td>
</tr>
}
+
+ private def renderMLCacheStatus(session: SessionInfo): Seq[Node] = {
+ if (session.finishTimestamp > 0 || mlCacheStatuses.isEmpty) {
+ <span>N/A</span>
+ } else {
+ mlCacheStatuses
+ .flatMap(_.get(SessionKey(session.userId, session.sessionId)))
+ .filter(_.models.nonEmpty)
+ .map { status =>
+ if (status.memoryControlEnabled) {
+ val inMemoryModels = status.models.count(_.inMemory)
+ val objectLabel = if (inMemoryModels == 1) "object" else "objects"
+ <span>
+ {s"$inMemoryModels $objectLabel in memory"}<br/>
+ {
+ s"${Utils.bytesToString(status.inMemorySizeBytes)} / " +
+ s"${Utils.bytesToString(status.maxInMemorySizeBytes)} memory"
+ }<br/>
+ {
+ s"${Utils.bytesToString(status.totalSizeBytes)} / " +
+ s"${Utils.bytesToString(status.maxTotalSizeBytes)} total"
+ }
+ </span>
+ } else {
+ val cachedModels = status.models.size
+ val objectLabel = if (cachedModels == 1) "object" else "objects"
+ <span>
+ Memory control disabled<br/>
+ {s"$cachedModels cached $objectLabel"}
+ </span>
+ }
+ }
+ .getOrElse(<span>Not used</span>)
+ }
+ }
+}
+
+private[ui] case class MLCacheModelTableRow(
+ userId: String,
+ sessionId: String,
+ model: MLCacheModelInfo)
+
+private[ui] class MLCacheModelStatsPagedTable(
+ request: HttpServletRequest,
+ parent: SparkConnectServerTab,
+ data: Seq[MLCacheModelTableRow],
+ subPath: String,
+ basePath: String,
+ tableTag: String)
+ extends PagedTable[MLCacheModelTableRow] {
+
+ private val (sortColumn, desc, pageSize) =
+ getTableParameters(request, tableTag, "Estimated Size")
+
+ private val encodedSortColumn = URLEncoder.encode(sortColumn, UTF_8.name())
+ private val parameterPath = s"$basePath/$subPath/?${getParameterOtherTable(request, tableTag)}"
+
+ override val dataSource =
+ new MLCacheModelTableDataSource(data, pageSize, sortColumn, desc)
+
+ override def tableId: String = tableTag
+
+ override def tableCssClass: String =
+ "table table-bordered table-sm table-striped table-head-clickable table-cell-width-limited"
+
+ override def pageLink(page: Int): String = {
+ parameterPath +
+ s"&$pageNumberFormField=$page" +
+ s"&$tableTag.sort=$encodedSortColumn" +
+ s"&$tableTag.desc=$desc" +
+ s"&$pageSizeFormField=$pageSize" +
+ s"#$tableTag"
+ }
+
+ override def pageSizeFormField: String = s"$tableTag.pageSize"
+
+ override def pageNumberFormField: String = s"$tableTag.page"
+
+ override def goButtonFormPath: String =
+ s"$parameterPath&$tableTag.sort=$encodedSortColumn" +
+ s"&$tableTag.desc=$desc#$tableTag"
+
+ override def headers: Seq[Node] = {
+ val headersAndTooltips: Seq[(String, Boolean, Option[String])] = Seq(
+ ("User", true, None),
+ ("Session ID", true, None),
+ ("Model ID", true, None),
+ ("Model Class", true, None),
+ ("Model Details", true, Some(SPARK_CONNECT_ML_CACHE_MODEL_DETAILS)),
+ ("Estimated Size", true, Some(SPARK_CONNECT_ML_CACHE_ESTIMATED_SIZE)),
+ ("Storage", true, Some(SPARK_CONNECT_ML_CACHE_STORAGE)))
+
+ isSortColumnValid(headersAndTooltips, sortColumn)
+ headerRow(headersAndTooltips, desc, pageSize, sortColumn, parameterPath, tableTag, tableTag)
+ }
+
+ override def row(row: MLCacheModelTableRow): Seq[Node] = {
+ val model = row.model
+ val sessionLink = "%s/%s/session/?id=%s&userId=%s".format(
+ UIUtils.prependBaseUri(request, parent.basePath),
+ parent.prefix,
+ URLEncoder.encode(row.sessionId, UTF_8.name()),
+ ConnectUiUtils.encodeUserId(row.userId))
+ <tr>
+ <td>{row.userId}</td>
+ <td><a href={sessionLink}>{row.sessionId}</a></td>
+ <td>{model.id}</td>
+ <td>{model.className}</td>
+ <td>{model.modelString}</td>
+ <td>{model.estimatedSizeBytes.map(Utils.bytesToString).getOrElse("N/A")}</td>
+ <td>{if (model.inMemory) "In memory" else "Offloaded"}</td>
+ </tr>
+ }
+}
+
+private[ui] class MLCacheModelTableDataSource(
+ info: Seq[MLCacheModelTableRow],
+ pageSize: Int,
+ sortColumn: String,
+ desc: Boolean)
+ extends PagedDataSource[MLCacheModelTableRow](pageSize) {
+
+ private val data = info.sorted(ordering(sortColumn, desc))
+
+ override def dataSize: Int = data.size
+
+ override def sliceData(from: Int, to: Int): Seq[MLCacheModelTableRow] = data.slice(from, to)
+
+ private def ordering(sortColumn: String, desc: Boolean): Ordering[MLCacheModelTableRow] = {
+ val ordering: Ordering[MLCacheModelTableRow] = sortColumn match {
+ case "User" => Ordering.by(_.userId)
+ case "Session ID" => Ordering.by(_.sessionId)
+ case "Model ID" => Ordering.by(_.model.id)
+ case "Model Class" => Ordering.by(_.model.className)
+ case "Model Details" => Ordering.by(_.model.modelString)
+ case "Estimated Size" =>
+ Ordering.by((row: MLCacheModelTableRow) =>
+ (row.model.estimatedSizeBytes.isDefined, row.model.estimatedSizeBytes.getOrElse(0L)))
Review Comment:
Addressed in c5c91ba9ac2. Estimated Size now orders directly by Option[Long], preserving None-before-Some without tuple allocation.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:
##########
@@ -179,6 +181,80 @@ private[ui] class SparkConnectServerPage(parent: SparkConnectServerTab)
content
}
+
+ /** Generate live ML cache statistics for active Spark Connect sessions. */
+ private def generateMLCacheStatsTable(request: HttpServletRequest): Seq[Node] = {
+ val cacheStatuses = parent.getMLCacheStatuses.filter(_._2.models.nonEmpty)
Review Comment:
Addressed in c5c91ba9ac2. The manager now returns every active session with an optional snapshot, so an uninitialized or empty cache renders Not used, while unavailable or removed live state renders N/A.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -280,4 +313,40 @@ private[connect] class MLCache(sessionHolder: SessionHolder) extends Logging {
}
info.result()
}
+
+ /** Returns a cache snapshot without loading or touching any cached model. */
+ def getStatus: MLCacheStatus = this.synchronized {
+ val models = mutable.ArrayBuilder.make[MLCacheModelInfo]
+ cachedModelMetadata.asScala.foreach { case (id, metadata) =>
+ models += MLCacheModelInfo(
+ id = id,
+ className = metadata.className,
+ modelString = metadata.modelString,
+ estimatedSizeBytes = metadata.estimatedSizeBytes,
+ inMemory = inMemoryModelIds.contains(id))
+ }
+ MLCacheStatus(
+ memoryControlEnabled = getMemoryControlEnabled,
+ inMemorySizeBytes = totalMLCacheInMemorySizeBytes.get(),
+ maxInMemorySizeBytes = sessionHolder.session.conf.get(
+ Connect.CONNECT_SESSION_CONNECT_ML_CACHE_MEMORY_CONTROL_MAX_IN_MEMORY_SIZE),
+ totalSizeBytes = totalMLCacheSizeBytes.get(),
+ maxTotalSizeBytes = getMLCacheMaxSize,
+ models = models.result().toIndexedSeq.sortBy(_.id))
Review Comment:
Addressed in c5c91ba9ac2. getStatus now returns the metadata iteration directly without per-session sorting; the UI data source remains responsible for display ordering.
##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala:
##########
@@ -134,6 +140,118 @@ class SparkConnectServerPageSuite
" data-bs-target=\"#aggregated-sqlsessionstat\""))
}
+ test("Spark Connect Server page should show live ML cache statistics and model details") {
+ val store = getStatusStore(closeSession = false)
+
+ val request = mock(classOf[HttpServletRequest])
+ val tab = mock(classOf[SparkConnectServerTab], RETURNS_SMART_NULLS)
+ when(tab.startTime).thenReturn(Calendar.getInstance().getTime)
+ when(tab.store).thenReturn(store)
+ when(tab.appName).thenReturn("testing")
+ when(tab.headerTabs).thenReturn(Seq.empty)
+ when(tab.hasLiveMLCacheStatus).thenReturn(true)
+ when(tab.getMLCacheStatuses).thenReturn(
Review Comment:
Addressed in c5c91ba9ac2. Added a focused test using the production SessionManager -> SessionHolder -> MLCache -> SparkConnectServerTab -> page path. It verifies that rendering neither initializes an unused cache nor updates last-access time, both before and after explicit cache initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57754:
URL: https://github.com/apache/spark/pull/57754#discussion_r3737582982
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:
##########
@@ -179,6 +181,80 @@ private[ui] class SparkConnectServerPage(parent: SparkConnectServerTab)
content
}
+
+ /** Generate live ML cache statistics for active Spark Connect sessions. */
+ private def generateMLCacheStatsTable(request: HttpServletRequest): Seq[Node] = {
+ val cacheStatuses = parent.getMLCacheStatuses.filter(_._2.models.nonEmpty)
Review Comment:
This filter prevents the UI from showing the promised unused-cache state: before the first model is registered, or after the cache is cleared, the model list is empty and the whole section disappears. Please retain session-level statuses with empty model lists and render `Not used`; the no-live-manager path should likewise render the documented unavailable state instead of silently omitting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57754:
URL: https://github.com/apache/spark/pull/57754#discussion_r3748463747
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -280,4 +313,40 @@ private[connect] class MLCache(sessionHolder: SessionHolder) extends Logging {
}
info.result()
}
+
+ /** Returns a cache snapshot without loading or touching any cached model. */
+ def getStatus: MLCacheStatus = this.synchronized {
+ val models = mutable.ArrayBuilder.make[MLCacheModelInfo]
+ cachedModelMetadata.asScala.foreach { case (id, metadata) =>
+ models += MLCacheModelInfo(
+ id = id,
+ className = metadata.className,
+ modelString = metadata.modelString,
+ estimatedSizeBytes = metadata.estimatedSizeBytes,
+ inMemory = inMemoryModelIds.contains(id))
+ }
+ MLCacheStatus(
+ memoryControlEnabled = getMemoryControlEnabled,
+ inMemorySizeBytes = totalMLCacheInMemorySizeBytes.get(),
+ maxInMemorySizeBytes = sessionHolder.session.conf.get(
+ Connect.CONNECT_SESSION_CONNECT_ML_CACHE_MEMORY_CONTROL_MAX_IN_MEMORY_SIZE),
+ totalSizeBytes = totalMLCacheSizeBytes.get(),
+ maxTotalSizeBytes = getMLCacheMaxSize,
+ models = models.result().toIndexedSeq.sortBy(_.id))
Review Comment:
This per-session sort is discarded by the UI: `generateMLCacheStatsTable` concatenates all session snapshots, and `MLCacheModelTableDataSource` sorts the complete row set again using the requested column. Please return the snapshot without `sortBy` so each refresh performs only the display ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57754:
URL: https://github.com/apache/spark/pull/57754#discussion_r3751227032
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:
##########
@@ -431,8 +521,152 @@ private[ui] class SessionStatsPagedTable(
<td> {if (session.finishTimestamp > 0) formatDate(session.finishTimestamp)} </td>
<td> {formatDurationVerbose(session.totalTime)} </td>
<td> {session.totalExecution.toString} </td>
+ <td> {renderMLCacheStatus(session)} </td>
</tr>
}
+
+ private def renderMLCacheStatus(session: SessionInfo): Seq[Node] = {
+ if (session.finishTimestamp > 0 || mlCacheStatuses.isEmpty) {
+ <span>N/A</span>
+ } else {
+ mlCacheStatuses
+ .flatMap(_.get(SessionKey(session.userId, session.sessionId)))
+ .filter(_.models.nonEmpty)
+ .map { status =>
+ if (status.memoryControlEnabled) {
+ val inMemoryModels = status.models.count(_.inMemory)
+ val objectLabel = if (inMemoryModels == 1) "object" else "objects"
+ <span>
+ {s"$inMemoryModels $objectLabel in memory"}<br/>
+ {
+ s"${Utils.bytesToString(status.inMemorySizeBytes)} / " +
+ s"${Utils.bytesToString(status.maxInMemorySizeBytes)} memory"
+ }<br/>
+ {
+ s"${Utils.bytesToString(status.totalSizeBytes)} / " +
+ s"${Utils.bytesToString(status.maxTotalSizeBytes)} total"
+ }
+ </span>
+ } else {
+ val cachedModels = status.models.size
+ val objectLabel = if (cachedModels == 1) "object" else "objects"
+ <span>
+ Memory control disabled<br/>
+ {s"$cachedModels cached $objectLabel"}
+ </span>
+ }
+ }
+ .getOrElse(<span>Not used</span>)
+ }
+ }
+}
+
+private[ui] case class MLCacheModelTableRow(
+ userId: String,
+ sessionId: String,
+ model: MLCacheModelInfo)
+
+private[ui] class MLCacheModelStatsPagedTable(
+ request: HttpServletRequest,
+ parent: SparkConnectServerTab,
+ data: Seq[MLCacheModelTableRow],
+ subPath: String,
+ basePath: String,
+ tableTag: String)
+ extends PagedTable[MLCacheModelTableRow] {
+
+ private val (sortColumn, desc, pageSize) =
+ getTableParameters(request, tableTag, "Estimated Size")
+
+ private val encodedSortColumn = URLEncoder.encode(sortColumn, UTF_8.name())
+ private val parameterPath = s"$basePath/$subPath/?${getParameterOtherTable(request, tableTag)}"
+
+ override val dataSource =
+ new MLCacheModelTableDataSource(data, pageSize, sortColumn, desc)
+
+ override def tableId: String = tableTag
+
+ override def tableCssClass: String =
+ "table table-bordered table-sm table-striped table-head-clickable table-cell-width-limited"
+
+ override def pageLink(page: Int): String = {
+ parameterPath +
+ s"&$pageNumberFormField=$page" +
+ s"&$tableTag.sort=$encodedSortColumn" +
+ s"&$tableTag.desc=$desc" +
+ s"&$pageSizeFormField=$pageSize" +
+ s"#$tableTag"
+ }
+
+ override def pageSizeFormField: String = s"$tableTag.pageSize"
+
+ override def pageNumberFormField: String = s"$tableTag.page"
+
+ override def goButtonFormPath: String =
+ s"$parameterPath&$tableTag.sort=$encodedSortColumn" +
+ s"&$tableTag.desc=$desc#$tableTag"
+
+ override def headers: Seq[Node] = {
+ val headersAndTooltips: Seq[(String, Boolean, Option[String])] = Seq(
+ ("User", true, None),
+ ("Session ID", true, None),
+ ("Model ID", true, None),
+ ("Model Class", true, None),
+ ("Model Details", true, Some(SPARK_CONNECT_ML_CACHE_MODEL_DETAILS)),
+ ("Estimated Size", true, Some(SPARK_CONNECT_ML_CACHE_ESTIMATED_SIZE)),
+ ("Storage", true, Some(SPARK_CONNECT_ML_CACHE_STORAGE)))
+
+ isSortColumnValid(headersAndTooltips, sortColumn)
+ headerRow(headersAndTooltips, desc, pageSize, sortColumn, parameterPath, tableTag, tableTag)
+ }
+
+ override def row(row: MLCacheModelTableRow): Seq[Node] = {
+ val model = row.model
+ val sessionLink = "%s/%s/session/?id=%s&userId=%s".format(
+ UIUtils.prependBaseUri(request, parent.basePath),
+ parent.prefix,
+ URLEncoder.encode(row.sessionId, UTF_8.name()),
+ ConnectUiUtils.encodeUserId(row.userId))
+ <tr>
+ <td>{row.userId}</td>
+ <td><a href={sessionLink}>{row.sessionId}</a></td>
+ <td>{model.id}</td>
+ <td>{model.className}</td>
+ <td>{model.modelString}</td>
+ <td>{model.estimatedSizeBytes.map(Utils.bytesToString).getOrElse("N/A")}</td>
+ <td>{if (model.inMemory) "In memory" else "Offloaded"}</td>
+ </tr>
+ }
+}
+
+private[ui] class MLCacheModelTableDataSource(
+ info: Seq[MLCacheModelTableRow],
+ pageSize: Int,
+ sortColumn: String,
+ desc: Boolean)
+ extends PagedDataSource[MLCacheModelTableRow](pageSize) {
+
+ private val data = info.sorted(ordering(sortColumn, desc))
+
+ override def dataSize: Int = data.size
+
+ override def sliceData(from: Int, to: Int): Seq[MLCacheModelTableRow] = data.slice(from, to)
+
+ private def ordering(sortColumn: String, desc: Boolean): Ordering[MLCacheModelTableRow] = {
+ val ordering: Ordering[MLCacheModelTableRow] = sortColumn match {
+ case "User" => Ordering.by(_.userId)
+ case "Session ID" => Ordering.by(_.sessionId)
+ case "Model ID" => Ordering.by(_.model.id)
+ case "Model Class" => Ordering.by(_.model.className)
+ case "Model Details" => Ordering.by(_.model.modelString)
+ case "Estimated Size" =>
+ Ordering.by((row: MLCacheModelTableRow) =>
+ (row.model.estimatedSizeBytes.isDefined, row.model.estimatedSizeBytes.getOrElse(0L)))
Review Comment:
This comparator allocates a tuple on every key extraction throughout the sort. Ordering directly by `row.model.estimatedSizeBytes` preserves the current `None`-before-`Some` order and avoids those repeated temporary allocations.
##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala:
##########
@@ -134,6 +140,118 @@ class SparkConnectServerPageSuite
" data-bs-target=\"#aggregated-sqlsessionstat\""))
}
+ test("Spark Connect Server page should show live ML cache statistics and model details") {
+ val store = getStatusStore(closeSession = false)
+
+ val request = mock(classOf[HttpServletRequest])
+ val tab = mock(classOf[SparkConnectServerTab], RETURNS_SMART_NULLS)
+ when(tab.startTime).thenReturn(Calendar.getInstance().getTime)
+ when(tab.store).thenReturn(store)
+ when(tab.appName).thenReturn("testing")
+ when(tab.headerTabs).thenReturn(Seq.empty)
+ when(tab.hasLiveMLCacheStatus).thenReturn(true)
+ when(tab.getMLCacheStatuses).thenReturn(
Review Comment:
Could this exercise the production `SessionManager -> SessionHolder -> MLCache -> page` handoff instead of stubbing `getMLCacheStatuses`? That focused integration test should also assert that rendering neither refreshes the session's last-access time nor initializes an unused cache; the current cache and page tests cover only the two ends independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5215975600
Thank you @zhengruifeng!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][ML][CONNECT] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5177839419
<img width="2107" height="1257" alt="image" src="https://github.com/user-attachments/assets/9ae83c94-98d5-4ab1-9e37-ec4baeda47d3" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5253216212
@dongjoon-hyun Thanks for the detailed review. Addressed in c5c91ba9ac2:
- The PR description now calls out the offloaded-model cleanup/removal semantics and states that the status covers models only (`Summary` objects are excluded).
- Live ML cache snapshots are collected before entering `store.synchronized`.
- The live-status API is now a single optional map containing each active session's optional snapshot, which distinguishes unavailable/removed state (`N/A`) from an uninitialized cache (`Not used`).
- `getStatus` now builds the metadata snapshot directly without the `ArrayBuilder`.
- A focused production-path test covers `SessionManager -> SessionHolder -> MLCache -> SparkConnectServerTab -> page`, including the last-access-time and lazy-initialization assertions.
The soft-value removal-listener timing remains an accepted transient limitation. The focused page suite passes all 7 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][ML][CONNECT] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5177618305
<img width="2109" height="1309" alt="image" src="https://github.com/user-attachments/assets/59c2e447-19b1-4ab9-af0a-6f68d176fc94" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5275520449
**Merge Summary:**
- merged into branch-4.x https://github.com/apache/spark/commit/6c1dc7e8e4bed5c73899e34cc4ddb4eb076e6632
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #57754: [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI
URL: https://github.com/apache/spark/pull/57754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][ML][CONNECT] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5176863470
Here is a simple end-to-end PySpark ML example for manually checking the UI.
Start a Connect server from the built Spark checkout:
```bash
./sbin/start-connect-server.sh --master local[2]
```
In another terminal, start a Connect PySpark shell:
```bash
./bin/pyspark --remote sc://localhost:15002
```
Then run:
```python
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.linalg import Vectors
training = spark.createDataFrame(
[
(1.0, Vectors.dense([0.0, 1.1, 0.1])),
(0.0, Vectors.dense([2.0, 1.0, -1.0])),
(0.0, Vectors.dense([2.0, 1.3, 1.0])),
(1.0, Vectors.dense([0.0, 1.2, -0.5])),
],
["label", "features"],
)
model = LogisticRegression(maxIter=5, regParam=0.01).fit(training)
model.transform(training).select("label", "prediction", "probability").show(
truncate=False
)
```
Keep the PySpark shell open and visit `http://localhost:4040/connect/` (or the Spark UI URL printed by the server). Before `fit`, the session's **ML Cache** cell should say `Not used`. After `fit`, reload the page; it should show one in-memory object and nonzero in-memory/total cache usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5212493903
also cc @haoyangeng-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5274634037
latest ui is like:
<img width="2230" height="1160" alt="image" src="https://github.com/user-attachments/assets/6dcd1a90-0302-4d2b-9a42-225b5d20731c" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58556][CONNECT][UI] Show ML cache status in Spark Connect UI [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5275509863
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/2f4bd74e30f5e6b807dd6aa993d1391efc809496
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org