You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/10/12 21:35:01 UTC

(spark) branch branch-4.3 updated: [SPARK-60158][UI] Avoid per-character synchronized reads in the bounded event log line reader

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new a4c63cb52722 [SPARK-60158][UI] Avoid per-character synchronized reads in the bounded event log line reader
a4c63cb52722 is described below

commit a4c63cb52722e15cb219606ec4f50c709bc03eac
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Mon Oct 12 14:34:33 2026 -0700

    [SPARK-60158][UI] Avoid per-character synchronized reads in the bounded event log line reader
    
    ### What changes were proposed in this pull request?
    
    The bounded line reader used by `ReplayListenerBus.replay(InputStream, ...)` (added in SPARK-59407, byte accounting fixed in SPARK-59804) called `BufferedReader.read()` once per character. This PR makes it read decoded characters in bulk instead:
    
    - `boundedLines` reads from the `InputStreamReader` into an 8192-char array, instead of wrapping the reader in a `BufferedReader` and calling `read()` per character. `BufferedReader.read()` takes a lock on every call.
    - `BoundedLineBuffer.append(c)` is replaced by `appendUntilNewline(chars, start, end)`, which finds the next `\n` and adds up the UTF-8 length in a single pass, appends the range in one call if it fits, and returns the index of the `\n`. Once the line exceeds the allowance, it releases the retained prefix and only scans for `\n` from then on.
    
    The reader's semantics are unchanged: the UTF-8 content byte limit with the trailing-CR allowance, physical line indices including skipped lines, strict decoding, and releasing the buffer of an over-long line while it is drained. The `Iterator[String]` overload of `replay` is not affected.
    
    ### Why are the changes needed?
    
    Since SPARK-59407, line splitting in this reader is several times slower than the previous `Source.getLines()`/`readLine()` (measured at about 14x in the review of #59074). Every replay through the `InputStream` overload pays this: loading application UIs in the History Server, event log compaction, and listing with fast in-progress parsing.
    
    It also matters for #59327 (SPARK-60109), which routes the History Server end-event reparse through this reader. For compressed logs, that reparse reads most of the decompressed file, because the skip target is computed from the compressed length, so without this change the listing of completed applications would become several times slower.
    
    A local measurement of a full replay through `replay(InputStream, ...)` with the History Server listing filter (210 MB of uncompressed event lines built from `spark-events/local-1642039451826` with randomized digits, median of 3 runs, JDK 21):
    
    | codec | file size | decompression only | `Source.getLines()` + `replay(Iterator)` | bounded reader before | bounded reader after |
    |---|---|---|---|---|---|
    | lz4 | 56 MB | 0.08 s | 0.21 s | 1.13 s | 0.22 s |
    | zstd | 16 MB | 0.06 s | 0.17-0.18 s | 1.10 s | 0.20 s |
    
    ### Does this PR introduce _any_ user-facing change?
    
    No behavior change. Replaying event logs through the bounded reader (History Server UI loading, listing and event log compaction) is faster.
    
    ### How was this patch tested?
    
    New tests in `ReplayListenerSuite`:
    - `SPARK-60158: Line buffer gives the same result for any split of a line`: random lines (ASCII, CR, 2-, 3- and 4-byte characters) split at random points; the returned `\n` index and the result match a reference that drops one trailing CR and keeps the line if its UTF-8 length fits, and the buffer is released once the line exceeds the limit plus the CR allowance.
    - `SPARK-60158: Replay splits lines across read buffers like a reference splitter`: randomized inputs with lines up to 20000 characters, so that lines and multi-byte characters cross the 8192-char read buffer, compared with a reference that splits on `\n`, drops one trailing CR and keeps lines within the byte limit. Each input is replayed both from a `ByteArrayInputStream` and from a stream that returns 1 to N bytes per read with `available() == 0`, like codec streams.
    - `SPARK-60158: Replay reports physical line numbers for lines spanning read buffers`: a skipped 20000-char line followed by a malformed line, with full and short reads; the skip warning and `Malformed line #N` report the physical line numbers.
    
    The existing SPARK-59804 tests (byte limit grid across the read buffer boundary, physical line numbers, skipped line diagnostics, malformed UTF-8 and stream IO errors) pass unchanged. The SPARK-59804 line buffer test now uses `appendUntilNewline`. `ReplayListenerSuite`, `FsHistoryProviderSuite` (RocksDB backend), `EventLogFileCompactorSuite`, `core/scalastyle` and `core/Test/scalastyle` pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59360 from viirya/SPARK-60158.
    
    Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
    (cherry picked from commit a2659a761e96e13e5785f68b9639d6a4c4818434)
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
 .../apache/spark/scheduler/ReplayListenerBus.scala |  71 ++++++++++---
 .../spark/scheduler/ReplayListenerSuite.scala      | 117 ++++++++++++++++++++-
 2 files changed, 170 insertions(+), 18 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
index e2d7892428f6..69720567d34d 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
@@ -17,7 +17,7 @@
 
 package org.apache.spark.scheduler
 
-import java.io.{BufferedReader, EOFException, InputStream, InputStreamReader, IOException}
+import java.io.{EOFException, InputStream, InputStreamReader, IOException}
 import java.nio.charset.{CodingErrorAction, StandardCharsets}
 
 import scala.annotation.tailrec
@@ -89,13 +89,27 @@ private[spark] class ReplayListenerBus(
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)
       .onUnmappableCharacter(CodingErrorAction.REPORT)
-    val reader = new BufferedReader(new InputStreamReader(logData, decoder))
+    // Read decoded characters in bulk and scan them for '\n', instead of calling read() per
+    // character on a BufferedReader, which takes a lock on every call.
+    val reader = new InputStreamReader(logData, decoder)
     new Iterator[(String, Int)] {
+      private val chars = new Array[Char](8192)
+      private var pos = 0
+      private var len = 0
       private var nextLine: (String, Int) = _
       private var lineIndex = 0
       private var lineFetched = false
       private var warned = false
 
+      /** Makes at least one unread character available; returns false at EOF. */
+      private def fill(): Boolean = {
+        while (pos == len && len != -1) {
+          len = reader.read(chars)
+          pos = 0
+        }
+        len != -1
+      }
+
       override def hasNext: Boolean = {
         if (!lineFetched) {
           nextLine = fetchLine()
@@ -116,15 +130,16 @@ private[spark] class ReplayListenerBus(
 
       @tailrec private def fetchLine(): (String, Int) = {
         val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
-        var c = reader.read()
-        if (c == -1) {
+        if (!fill()) {
           null
         } else {
           val index = lineIndex
           lineIndex += 1
-          while (c != -1 && c != '\n') {
-            buffer.append(c.toChar)
-            c = reader.read()
+          var endOfLine = false
+          while (!endOfLine && fill()) {
+            val i = buffer.appendUntilNewline(chars, pos, len)
+            endOfLine = i < len
+            pos = if (endOfLine) i + 1 else i
           }
           val maybeLine = buffer.result()
           if (maybeLine.isEmpty) {
@@ -273,19 +288,41 @@ private[spark] object ReplayListenerBus {
 
     def capacity: Int = buffer.capacity()
 
-    def append(c: Char): Unit = {
+    /**
+     * Appends `chars` from `start` up to the first '\n' before `end`, and returns the index of
+     * that '\n', or `end` if there is none. The line ending is found and the UTF-8 length is
+     * counted in a single pass. Once the line exceeds the allowance, its retained prefix is
+     * released and the rest of the line is only scanned for '\n'.
+     */
+    def appendUntilNewline(chars: Array[Char], start: Int, end: Int): Int = {
+      var i = start
+      // Reserve one extra byte for a possible CR in the line ending.
+      while (i < end && chars(i) != '\n' && byteLength <= bufferLimit) {
+        byteLength += utf8Width(chars(i))
+        i += 1
+      }
       if (byteLength <= bufferLimit) {
-        // The strict decoder emits valid surrogate pairs: count two bytes per half.
-        byteLength += (if (c < 0x80) 1 else if (c < 0x800 || Character.isSurrogate(c)) 2 else 3)
-        // Reserve one extra byte for a possible CR in the line ending.
-        if (byteLength <= bufferLimit) {
-          buffer.append(c)
-        } else {
-          // The line will be skipped; release the retained prefix while draining.
-          buffer.setLength(0)
-          buffer.trimToSize()
+        buffer.append(chars, start, i - start)
+      } else {
+        release()
+        while (i < end && chars(i) != '\n') {
+          i += 1
         }
       }
+      i
+    }
+
+    // The strict decoder emits valid surrogate pairs: count two bytes per half.
+    private def utf8Width(c: Char): Int = {
+      if (c < 0x80) 1 else if (c < 0x800 || Character.isSurrogate(c)) 2 else 3
+    }
+
+    // The line will be skipped; release the retained prefix while draining.
+    private def release(): Unit = {
+      if (buffer.capacity() > 0) {
+        buffer.setLength(0)
+        buffer.trimToSize()
+      }
     }
 
     def result(): Option[String] = {
diff --git a/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala b/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
index 882d3e711a02..a2802f3a8f70 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
@@ -158,8 +158,9 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
   test("SPARK-59804: Replay line buffer stops retaining oversized content") {
     for (limit <- Seq(0, 1, 8, 1024)) {
       val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      val x = Array('x')
       for (_ <- 0 until 10000) {
-        buffer.append('x')
+        assert(buffer.appendUntilNewline(x, 0, 1) == 1)
         assert(buffer.length <= limit + 1)
       }
       assert(buffer.length == 0)
@@ -207,6 +208,90 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(diagnostic.length < 1200)
   }
 
+  test("SPARK-60158: Line buffer gives the same result for any split of a line") {
+    val alphabet = Seq("x", "\r", 0x00e9.toChar.toString, 0x4e2d.toChar.toString,
+      new String(Character.toChars(0x1f600)))
+    val random = new scala.util.Random(60158)
+    for (_ <- 0 until 2000) {
+      val line = Seq.fill(random.nextInt(40))(alphabet(random.nextInt(alphabet.length))).mkString
+      val limit = random.nextInt(80)
+      // Reference: drop one trailing CR and keep the line if its UTF-8 length fits the limit.
+      val content = if (line.endsWith("\r")) line.dropRight(1) else line
+      val expected =
+        if (content.getBytes(StandardCharsets.UTF_8).length <= limit) Some(content) else None
+      // A '\n' and a following line after the end must not be consumed.
+      val chars = (line + "\nnext").toCharArray
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      var start = 0
+      var newline = -1
+      while (newline < 0) {
+        val end = math.min(chars.length, start + random.nextInt(8))
+        val i = buffer.appendUntilNewline(chars, start, end)
+        assert(i >= start && i <= end)
+        if (i < end) newline = i else start = end
+      }
+      withClue(s"line=${line.map(_.toInt).mkString(",")} limit=$limit: ") {
+        assert(newline == line.length)
+        assert(buffer.result() == expected)
+        // The buffer is released once the line exceeds the limit plus the CR allowance.
+        if (line.getBytes(StandardCharsets.UTF_8).length > limit + 1) {
+          assert(buffer.capacity == 0)
+        }
+      }
+    }
+  }
+
+  test("SPARK-60158: Replay splits lines across read buffers like a reference splitter") {
+    val alphabet = Seq("x", "\r", 0x00e9.toChar.toString, 0x4e2d.toChar.toString,
+      new String(Character.toChars(0x1f600)))
+    val random = new scala.util.Random(60158)
+    for (_ <- 0 until 50) {
+      val lines = Seq.fill(random.nextInt(20)) {
+        val length = if (random.nextInt(4) == 0) random.nextInt(20000) else random.nextInt(100)
+        Seq.fill(length)(alphabet(random.nextInt(alphabet.length))).mkString
+      }
+      val data = lines.mkString("\n") + (if (random.nextBoolean()) "\n" else "")
+      val bytes = data.getBytes(StandardCharsets.UTF_8)
+      val limit = 1 + random.nextInt(30000)
+      // Reference: split on '\n', drop one trailing CR, keep lines within the byte limit.
+      val expected = (if (data.isEmpty) Seq.empty else data.split("\n", -1).toSeq)
+        .dropRight(if (data.endsWith("\n")) 1 else 0)
+        .map(line => if (line.endsWith("\r")) line.dropRight(1) else line)
+        .filter(_.getBytes(StandardCharsets.UTF_8).length <= limit)
+      // Full reads, and the short, uneven reads of codec streams.
+      val maxRead = Seq(1, 3, 64, 8192)(random.nextInt(4))
+      for (input <- Seq(new ByteArrayInputStream(bytes),
+          new ShortReadInputStream(bytes, random, maxRead))) {
+        val seen = new ArrayBuffer[String]
+        assert(new ReplayListenerBus(limit).replay(input, "reference", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == expected, s"limit=$limit input=${input.getClass.getSimpleName}")
+      }
+    }
+  }
+
+  test("SPARK-60158: Replay reports physical line numbers for lines spanning read buffers") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    // Line 2 is skipped and spans several read buffers; line 4 is malformed.
+    val bytes = (end + "\n" + "x" * 20000 + "\n" + end + "\n{}\n")
+      .getBytes(StandardCharsets.UTF_8)
+    val random = new scala.util.Random(60158)
+    for (input <- Seq(new ByteArrayInputStream(bytes), new ShortReadInputStream(bytes, random, 1),
+        new ShortReadInputStream(bytes, random, 100))) {
+      val appender = new LogAppender
+      withLogAppender(appender) {
+        assert(!new ReplayListenerBus(1024).replay(input, "spanning"))
+      }
+      val messages = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+      withClue(s"input=${input.getClass.getSimpleName}: ") {
+        assert(messages.exists(_.contains("in spanning; first skipped line: 2")), messages)
+        assert(messages.exists(_.startsWith("Malformed line #4: {}")), messages)
+      }
+    }
+  }
+
   test("SPARK-59804: Replay line limit defaults and maximum supported configuration") {
     val conf = new SparkConf(false)
     assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
@@ -505,6 +590,36 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
    * This is a dummy input stream that wraps another input stream but ends prematurely when
    * reading at the specified position, throwing an EOFException.
    */
+  /** Returns 1 to `maxRead` bytes per read and reports nothing available, like codec streams. */
+  private class ShortReadInputStream(data: Array[Byte], random: scala.util.Random, maxRead: Int)
+    extends InputStream {
+    private var pos = 0
+
+    override def read(): Int = {
+      if (pos < data.length) {
+        pos += 1
+        data(pos - 1) & 0xFF
+      } else {
+        -1
+      }
+    }
+
+    override def read(b: Array[Byte], off: Int, len: Int): Int = {
+      if (len == 0) {
+        0
+      } else if (pos >= data.length) {
+        -1
+      } else {
+        val n = math.min(math.min(len, 1 + random.nextInt(maxRead)), data.length - pos)
+        System.arraycopy(data, pos, b, off, n)
+        pos += n
+        n
+      }
+    }
+
+    override def available(): Int = 0
+  }
+
   private class EarlyEOFInputStream(in: InputStream, failAtPos: Int) extends InputStream {
     private val countDown = new AtomicInteger(failAtPos)
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org