You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by gg...@apache.org on 2026/08/25 14:55:45 UTC

(livy) branch master updated: [LIVY-1070] Add LivySessionMetrics Codahale gauges for session monitoring

This is an automated email from the ASF dual-hosted git repository.

gyogal pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/livy.git


The following commit(s) were added to refs/heads/master by this push:
     new 92ccea35 [LIVY-1070] Add LivySessionMetrics Codahale gauges for session monitoring
92ccea35 is described below

commit 92ccea357d8ea2f8d8ecdcf8408b9de129e88adf
Author: samuhasavak5 <sa...@gmail.com>
AuthorDate: Tue Aug 25 20:25:39 2026 +0530

    [LIVY-1070] Add LivySessionMetrics Codahale gauges for session monitoring
    
    ## What changes were proposed in this pull request?
    
    This PR adds server-side session monitoring gauges for LIVY-1070.
    
    Problem: Livy exposes session state via the REST API (/sessions, /batches) but does not publish session counts as Codahale metrics on the existing /metrics endpoint. External monitoring systems must poll the REST API to observe session distribution.
    
    Solution: Introduce LivySessionMetrics, which registers session count gauges into Livy's MetricRegistry at server startup. Gauges are derived from InteractiveSessionManager and BatchSessionManager and exposed alongside existing Livy metrics via the AdminServlet at /metrics.
    
    Metrics registered (18 gauges)
    
    Overall (3)
    
        livy.sessions.total
        livy.sessions.active.total
        livy.sessions.terminal.total
    
    Interactive (8)
    
        livy.sessions.interactive.total
        livy.sessions.interactive.{idle,busy,starting,shutting_down,dead,error,killed}
    
    Batch (7)
    
        livy.sessions.batch.total
        livy.sessions.batch.{starting,running,success,dead,error,killed}
    
    Design notes
    
        Additive only — no REST API or session lifecycle behavior changes
        Idempotent registration — skips gauge names already present in the registry
        Error-safe callbacks — gauge getValue returns 0 on exception
        State normalization — handles case and hyphen/underscore variants (e.g. shuttingdown, succeeded)
        HA note — gauge values reflect the local Livy server instance; in HA deployments only the leader holds active sessions
    
    Compatibility
    
        No new endpoints; uses existing /metrics AdminServlet
        Backward compatible — new gauges appear alongside existing metrics
    
    ## How was this patch tested?
    
    Build
    mvn package -Pspark3 -Pscala-2.12 -pl server -am \
    -s /tmp/livy-mvn-central-settings.xml -DskipTests
    
    Result: BUILD SUCCESS
    Unit tests
    mvn test -Pspark3 -Pscala-2.12 -pl server \
    -s /tmp/livy-mvn-central-settings.xml \
    -Dsuites=org.apache.livy.server.LivySessionMetricsSpec
    
    Result: 8/8 tests passed
    
    Test | Coverage -- | -- All 18 gauge registrations | Registration Duplicate registration guard | Idempotency Zero sessions | Empty state Interactive sessions by state | State counting Batch sessions by state (incl. succeeded alias) | State counting Batch succeeded alias | Edge case Overall totals (total, active, terminal) | Aggregation Exception fallback returns 0 | Error handling
    Code coverage
    
    JaCoCo agent enabled during test run (server/target/jacoco/main.exec generated).
    
    No UI changes in this PR.
    
    ## Was this patch authored or co-authored using generative AI tooling?
    
    Yes, this was co-authored using Cursor to help generate the new test cases.
---
 docs/rest-api.md                                   | 101 ++++++++++
 .../scala/org/apache/livy/server/LivyServer.scala  |   2 +
 .../apache/livy/server/LivySessionMetrics.scala    | 142 ++++++++++++++
 .../livy/server/LivySessionMetricsSpec.scala       | 211 +++++++++++++++++++++
 4 files changed, 456 insertions(+)

diff --git a/docs/rest-api.md b/docs/rest-api.md
index fd614248..b4d2042d 100644
--- a/docs/rest-api.md
+++ b/docs/rest-api.md
@@ -593,6 +593,107 @@ Gets the log lines from this batch.
   </tr>
 </table>
 
+### GET /metrics/metrics
+
+Returns Codahale (Dropwizard) metrics for the Livy server, including session
+count gauges registered at startup. The metrics servlet is mounted at `/metrics`;
+the JSON payload is served at `/metrics/metrics`.
+
+Append `?pretty=true` for formatted output.
+
+#### Request Parameters
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>pretty</td>
+    <td>When set to <code>true</code>, returns formatted JSON</td>
+    <td>boolean</td>
+  </tr>
+</table>
+
+#### Response Body
+
+JSON object containing Codahale metrics. Session gauges are exposed under the
+`gauges` key. Each gauge reports the current session count on this Livy server
+instance.
+
+<table class="table">
+  <tr><th>Gauge name</th><th>Description</th></tr>
+  <tr>
+    <td>livy.sessions.total</td>
+    <td>Total interactive and batch sessions</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.active.total</td>
+    <td>Sessions in active states (interactive: starting, idle, busy; batch: starting, running)</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.terminal.total</td>
+    <td>Sessions in terminal states</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.total</td>
+    <td>Total interactive sessions</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.idle</td>
+    <td>Interactive sessions in <code>idle</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.busy</td>
+    <td>Interactive sessions in <code>busy</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.starting</td>
+    <td>Interactive sessions in <code>starting</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.shutting_down</td>
+    <td>Interactive sessions in <code>shutting_down</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.dead</td>
+    <td>Interactive sessions in <code>dead</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.error</td>
+    <td>Interactive sessions in <code>error</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.interactive.killed</td>
+    <td>Interactive sessions in <code>killed</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.total</td>
+    <td>Total batch sessions</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.starting</td>
+    <td>Batch sessions in <code>starting</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.running</td>
+    <td>Batch sessions in <code>running</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.success</td>
+    <td>Batch sessions in <code>success</code> or <code>succeeded</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.dead</td>
+    <td>Batch sessions in <code>dead</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.error</td>
+    <td>Batch sessions in <code>error</code> state</td>
+  </tr>
+  <tr>
+    <td>livy.sessions.batch.killed</td>
+    <td>Batch sessions in <code>killed</code> state</td>
+  </tr>
+</table>
+
 ## REST Objects
 
 ### Session
diff --git a/server/src/main/scala/org/apache/livy/server/LivyServer.scala b/server/src/main/scala/org/apache/livy/server/LivyServer.scala
index 73f259d5..68a3880a 100644
--- a/server/src/main/scala/org/apache/livy/server/LivyServer.scala
+++ b/server/src/main/scala/org/apache/livy/server/LivyServer.scala
@@ -252,6 +252,8 @@ class LivyServer extends Logging {
             }
 
             context.mountMetricsAdminServlet("/metrics")
+            LivySessionMetrics.register(
+              metricRegistry, interactiveSessionManager, batchSessionManager)
 
             mount(context, livyVersionServlet, "/version/*")
           } catch {
diff --git a/server/src/main/scala/org/apache/livy/server/LivySessionMetrics.scala b/server/src/main/scala/org/apache/livy/server/LivySessionMetrics.scala
new file mode 100644
index 00000000..0619f521
--- /dev/null
+++ b/server/src/main/scala/org/apache/livy/server/LivySessionMetrics.scala
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import com.codahale.metrics.{Gauge, MetricRegistry}
+
+import org.apache.livy.sessions.{BatchSessionManager, InteractiveSessionManager}
+
+/**
+ * Registers Livy session metrics as Codahale/Dropwizard gauges in the MetricRegistry.
+ *
+ * Uses the existing `io.dropwizard.metrics:metrics-core` dependency (Java package
+ * `com.codahale.metrics`) already declared in `server/pom.xml` and wired through
+ * `scalatra-metrics` (`MetricsBootstrap`) in `LivyServer`. No new metrics library
+ * is introduced by this class; it only adds session gauges to the shared registry
+ * exposed at `/metrics`.
+ *
+ * 18 gauges total:
+ *   - 3 overall (total, active_total, terminal_total)
+ *   - 8 interactive (total, idle, busy, starting, shutting_down, dead, error, killed)
+ *   - 7 batch (total, starting, running, success, dead, error, killed)
+ */
+object LivySessionMetrics {
+
+  /**
+   * Registers session count gauges into `metricRegistry`. Idempotent: existing gauge names
+   * are left unchanged. Gauge callbacks capture the session managers passed here, so no
+   * long-lived helper instance is required.
+   */
+  def register(
+      metricRegistry: MetricRegistry,
+      interactiveSessionManager: InteractiveSessionManager,
+      batchSessionManager: BatchSessionManager): Unit = {
+
+    def registerGauge(name: String, valueFn: => Int): Unit = {
+      if (!metricRegistry.getGauges.containsKey(name)) {
+        metricRegistry.register(name, new Gauge[Int] {
+          override def getValue: Int = {
+            try {
+              valueFn
+            } catch {
+              case _: Throwable => 0
+            }
+          }
+        })
+      }
+    }
+
+    def countInteractiveByState(expectedStates: Set[String]): Int = {
+      interactiveSessionManager.all().count { session =>
+        expectedStates.contains(normalizeState(session.state.toString))
+      }
+    }
+
+    def countBatchByState(expectedStates: Set[String]): Int = {
+      batchSessionManager.all().count { session =>
+        expectedStates.contains(normalizeState(session.state.toString))
+      }
+    }
+
+    def normalizeState(state: String): String = {
+      Option(state).getOrElse("").trim.toLowerCase.replace("-", "_").replace(" ", "_")
+    }
+
+    // ========== Interactive Session Metrics (8) ==========
+
+    registerGauge("livy.sessions.interactive.total",
+      interactiveSessionManager.size)
+
+    registerGauge("livy.sessions.interactive.idle",
+      countInteractiveByState(Set("idle")))
+
+    registerGauge("livy.sessions.interactive.starting",
+      countInteractiveByState(Set("starting")))
+
+    registerGauge("livy.sessions.interactive.busy",
+      countInteractiveByState(Set("busy")))
+
+    registerGauge("livy.sessions.interactive.dead",
+      countInteractiveByState(Set("dead")))
+
+    registerGauge("livy.sessions.interactive.shutting_down",
+      countInteractiveByState(Set("shutting_down", "shuttingdown")))
+
+    registerGauge("livy.sessions.interactive.error",
+      countInteractiveByState(Set("error")))
+
+    registerGauge("livy.sessions.interactive.killed",
+      countInteractiveByState(Set("killed")))
+
+    // ========== Batch Session Metrics (7) ==========
+
+    registerGauge("livy.sessions.batch.total",
+      batchSessionManager.size)
+
+    registerGauge("livy.sessions.batch.starting",
+      countBatchByState(Set("starting")))
+
+    registerGauge("livy.sessions.batch.running",
+      countBatchByState(Set("running")))
+
+    registerGauge("livy.sessions.batch.success",
+      countBatchByState(Set("success", "succeeded")))
+
+    registerGauge("livy.sessions.batch.dead",
+      countBatchByState(Set("dead")))
+
+    registerGauge("livy.sessions.batch.error",
+      countBatchByState(Set("error")))
+
+    registerGauge("livy.sessions.batch.killed",
+      countBatchByState(Set("killed")))
+
+    // ========== Overall Metrics (3) ==========
+
+    registerGauge("livy.sessions.total",
+      interactiveSessionManager.size + batchSessionManager.size)
+
+    registerGauge("livy.sessions.active.total",
+      countInteractiveByState(Set("starting", "idle", "busy")) +
+        countBatchByState(Set("starting", "running")))
+
+    registerGauge("livy.sessions.terminal.total",
+      countInteractiveByState(Set("dead", "error", "killed")) +
+        countBatchByState(Set("success", "succeeded", "dead", "error", "killed")))
+  }
+}
diff --git a/server/src/test/scala/org/apache/livy/server/LivySessionMetricsSpec.scala b/server/src/test/scala/org/apache/livy/server/LivySessionMetricsSpec.scala
new file mode 100644
index 00000000..25da6c28
--- /dev/null
+++ b/server/src/test/scala/org/apache/livy/server/LivySessionMetricsSpec.scala
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy.server
+
+import com.codahale.metrics.{Gauge, MetricRegistry}
+import org.mockito.Mockito._
+import org.scalatest.funspec.AnyFunSpec
+import org.scalatest.matchers.should.Matchers
+import org.scalatestplus.mockito.MockitoSugar
+
+import org.apache.livy.LivyBaseUnitTestSuite
+import org.apache.livy.server.batch.BatchSession
+import org.apache.livy.server.interactive.InteractiveSession
+import org.apache.livy.sessions.{BatchSessionManager, InteractiveSessionManager, SessionState}
+
+class LivySessionMetricsSpec extends AnyFunSpec with Matchers with MockitoSugar
+  with LivyBaseUnitTestSuite {
+
+  private val expectedGauges = Seq(
+    "livy.sessions.interactive.total",
+    "livy.sessions.interactive.idle",
+    "livy.sessions.interactive.starting",
+    "livy.sessions.interactive.busy",
+    "livy.sessions.interactive.dead",
+    "livy.sessions.interactive.shutting_down",
+    "livy.sessions.interactive.error",
+    "livy.sessions.interactive.killed",
+    "livy.sessions.batch.total",
+    "livy.sessions.batch.starting",
+    "livy.sessions.batch.running",
+    "livy.sessions.batch.success",
+    "livy.sessions.batch.dead",
+    "livy.sessions.batch.error",
+    "livy.sessions.batch.killed",
+    "livy.sessions.total",
+    "livy.sessions.active.total",
+    "livy.sessions.terminal.total"
+  )
+
+  private def gaugeValue(registry: MetricRegistry, name: String): Int = {
+    registry.getGauges.get(name).asInstanceOf[Gauge[Int]].getValue
+  }
+
+  private def mockInteractive(state: SessionState): InteractiveSession = {
+    val session = mock[InteractiveSession]
+    when(session.state).thenReturn(state)
+    session
+  }
+
+  private def mockBatch(state: SessionState): BatchSession = {
+    val session = mock[BatchSession]
+    when(session.state).thenReturn(state)
+    session
+  }
+
+  private def registerMetrics(
+      interactiveSessions: Seq[InteractiveSession] = Seq.empty,
+      batchSessions: Seq[BatchSession] = Seq.empty,
+      registry: MetricRegistry = new MetricRegistry()): Unit = {
+    val interactiveManager = mock[InteractiveSessionManager]
+    val batchManager = mock[BatchSessionManager]
+    when(interactiveManager.size).thenReturn(interactiveSessions.size)
+    when(interactiveManager.all()).thenReturn(interactiveSessions)
+    when(batchManager.size).thenReturn(batchSessions.size)
+    when(batchManager.all()).thenReturn(batchSessions)
+    LivySessionMetrics.register(registry, interactiveManager, batchManager)
+  }
+
+  describe("LivySessionMetrics") {
+
+    it("should register all 18 session gauges") {
+      val registry = new MetricRegistry()
+      registerMetrics(registry = registry)
+
+      expectedGauges.foreach { name =>
+        registry.getGauges.keySet() should contain(name)
+      }
+      registry.getGauges.size() shouldBe 18
+    }
+
+    it("should not register duplicate gauges on the same registry") {
+      val registry = new MetricRegistry()
+      val interactiveManager = mock[InteractiveSessionManager]
+      val batchManager = mock[BatchSessionManager]
+      when(interactiveManager.size).thenReturn(0)
+      when(interactiveManager.all()).thenReturn(Seq.empty)
+      when(batchManager.size).thenReturn(0)
+      when(batchManager.all()).thenReturn(Seq.empty)
+
+      LivySessionMetrics.register(registry, interactiveManager, batchManager)
+      LivySessionMetrics.register(registry, interactiveManager, batchManager)
+
+      registry.getGauges.size() shouldBe 18
+    }
+
+    it("should report zero when there are no sessions") {
+      val registry = new MetricRegistry()
+      registerMetrics(registry = registry)
+
+      expectedGauges.foreach { name =>
+        gaugeValue(registry, name) shouldBe 0
+      }
+    }
+
+    it("should count interactive sessions by state") {
+      val registry = new MetricRegistry()
+      val sessions = Seq(
+        mockInteractive(SessionState.Idle),
+        mockInteractive(SessionState.Idle),
+        mockInteractive(SessionState.Busy),
+        mockInteractive(SessionState.Starting),
+        mockInteractive(SessionState.ShuttingDown),
+        mockInteractive(SessionState.Dead()),
+        mockInteractive(SessionState.Error()),
+        mockInteractive(SessionState.Killed())
+      )
+      registerMetrics(interactiveSessions = sessions, registry = registry)
+
+      gaugeValue(registry, "livy.sessions.interactive.total") shouldBe 8
+      gaugeValue(registry, "livy.sessions.interactive.idle") shouldBe 2
+      gaugeValue(registry, "livy.sessions.interactive.busy") shouldBe 1
+      gaugeValue(registry, "livy.sessions.interactive.starting") shouldBe 1
+      gaugeValue(registry, "livy.sessions.interactive.shutting_down") shouldBe 1
+      gaugeValue(registry, "livy.sessions.interactive.dead") shouldBe 1
+      gaugeValue(registry, "livy.sessions.interactive.error") shouldBe 1
+      gaugeValue(registry, "livy.sessions.interactive.killed") shouldBe 1
+    }
+
+    it("should count batch sessions by state including succeeded alias") {
+      val registry = new MetricRegistry()
+      val sessions = Seq(
+        mockBatch(SessionState.Starting),
+        mockBatch(SessionState.Running),
+        mockBatch(SessionState.Success()),
+        mockBatch(SessionState.Dead()),
+        mockBatch(SessionState.Error()),
+        mockBatch(SessionState.Killed())
+      )
+      registerMetrics(batchSessions = sessions, registry = registry)
+
+      gaugeValue(registry, "livy.sessions.batch.total") shouldBe 6
+      gaugeValue(registry, "livy.sessions.batch.starting") shouldBe 1
+      gaugeValue(registry, "livy.sessions.batch.running") shouldBe 1
+      gaugeValue(registry, "livy.sessions.batch.success") shouldBe 1
+      gaugeValue(registry, "livy.sessions.batch.dead") shouldBe 1
+      gaugeValue(registry, "livy.sessions.batch.error") shouldBe 1
+      gaugeValue(registry, "livy.sessions.batch.killed") shouldBe 1
+    }
+
+    it("should count batch success for succeeded state alias") {
+      val registry = new MetricRegistry()
+      val succeededState = mock[SessionState]
+      when(succeededState.toString).thenReturn("succeeded")
+      val session = mock[BatchSession]
+      when(session.state).thenReturn(succeededState)
+      registerMetrics(batchSessions = Seq(session), registry = registry)
+
+      gaugeValue(registry, "livy.sessions.batch.success") shouldBe 1
+    }
+
+    it("should compute overall totals across interactive and batch sessions") {
+      val registry = new MetricRegistry()
+      val interactive = Seq(
+        mockInteractive(SessionState.Idle),
+        mockInteractive(SessionState.Dead())
+      )
+      val batch = Seq(
+        mockBatch(SessionState.Running),
+        mockBatch(SessionState.Success())
+      )
+      registerMetrics(
+        interactiveSessions = interactive, batchSessions = batch, registry = registry)
+
+      gaugeValue(registry, "livy.sessions.total") shouldBe 4
+      gaugeValue(registry, "livy.sessions.active.total") shouldBe 2
+      gaugeValue(registry, "livy.sessions.terminal.total") shouldBe 2
+    }
+
+    it("should return zero from gauges when session managers throw") {
+      val registry = new MetricRegistry()
+      val interactiveManager = mock[InteractiveSessionManager]
+      val batchManager = mock[BatchSessionManager]
+      when(interactiveManager.size).thenReturn(1)
+      when(interactiveManager.all()).thenThrow(new RuntimeException("interactive failure"))
+      when(batchManager.size).thenReturn(1)
+      when(batchManager.all()).thenThrow(new RuntimeException("batch failure"))
+
+      LivySessionMetrics.register(registry, interactiveManager, batchManager)
+
+      gaugeValue(registry, "livy.sessions.interactive.idle") shouldBe 0
+      gaugeValue(registry, "livy.sessions.batch.running") shouldBe 0
+      gaugeValue(registry, "livy.sessions.active.total") shouldBe 0
+      gaugeValue(registry, "livy.sessions.terminal.total") shouldBe 0
+    }
+  }
+}