You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/10/13 16:29:00 UTC

(spark) branch branch-4.2 updated: [SPARK-60109][UI] Apply the replay line-length limit and UTF-8-safe line alignment to the History Server end-event reparse

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new c97be2dd7b39 [SPARK-60109][UI] Apply the replay line-length limit and UTF-8-safe line alignment to the History Server end-event reparse
c97be2dd7b39 is described below

commit c97be2dd7b39d5be9883926923b60a6b5eabfc2a
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Tue Oct 13 09:28:18 2026 -0700

    [SPARK-60109][UI] Apply the replay line-length limit and UTF-8-safe line alignment to the History Server end-event reparse
    
    ### What changes were proposed in this pull request?
    
    When building the application listing, `FsHistoryProvider` looks for the application end event by reopening the last event log file, skipping to `len - spark.history.fs.endEventReparseChunkSize` bytes and re-parsing the rest. This PR changes how that tail is read:
    
    - The new `FsHistoryProvider.skipToLineStart` positions the stream at the first line that starts at or after the skip target, working on raw bytes before any UTF-8 decoding. `0x0A` never occurs inside a multi-byte UTF-8 sequence, so this is safe at any byte offset.
      - It skips to `target - 1` and scans from there, so a line that starts exactly at `target` (possibly the end event itself) is kept instead of being discarded.
      - When `skip` returns 0, it reads one byte to tell EOF apart from a short skip, so it terminates when the decompressed stream is shorter than `target` (e.g. when the compressed length exceeds the decompressed one).
      - The partial line is scanned in bulk; bytes read past the line boundary are put back in front of the stream with a `SequenceInputStream`.
    - The tail is then replayed through the bounded `InputStream` reader instead of the `Iterator[String]` overload fed by `Source.fromInputStream(in)(Codec.UTF8).getLines()`, so `spark.history.fs.eventLog.maxLineLength` and the strict decoder apply on this path as well.
    - The new `ReplayListenerBus.replayFromOffset` takes the byte offset of the stream's first line. When it is positive, every replay diagnostic that prints a line number (JSON errors, `Malformed line #N`, the truncation warning and the over-long line warning) adds `(lines counted from uncompressed byte offset N)`, with the offset logged under its own `OFFSET` key. The source name stays the plain file path, so the `PATH` / `FILE_NAME` fields are unchanged. The existing `replay` signature [...]
    
    `maybeTruncated`, `eventsFilter` and `HaltReplayException` handling are unchanged, since all `replay` variants share `replayEntries`. Without a skip (`target <= 0`), replay starts at offset 0 and the diagnostics are as before.
    
    ### Why are the changes needed?
    
    This is the tail-reparse follow-up discussed in #59074. With the defaults (completed apps, `endEventReparseChunkSize=1m`), every completed application takes this path, and because the target is computed from the compressed file length while the skip runs on the decompressed stream, most of a compressed log can be read there.
    
    1. The `Iterator[String]` overload bypasses the bounded line reader (SPARK-59407 / SPARK-59804), so `maxLineLength` did not apply and an over-long line was fully materialized.
    2. If the skip target landed inside a multi-byte UTF-8 character (e.g. CJK SQL text or job descriptions, which Jackson writes as raw UTF-8), `source.next()` threw `MalformedInputException` before replay started. `mergeApplicationListing` only logs it, and since the placeholder listing entry was already written with the same file size, the log is not retried, so the completed application stayed missing from the listing. The target is deterministic, so restarting the History Server fail [...]
    3. Line numbers in the replay diagnostics were relative to the skip point but printed next to the full file path.
    
    This builds on #59360 (SPARK-60158, merged), which makes the bounded reader read in bulk; without it, routing the tail through the bounded reader would make this step several times slower. The two should be backported together.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in the History Server:
    - A completed application whose end-event reparse offset lands inside a multi-byte UTF-8 character now appears in the application listing instead of being dropped.
    - `spark.history.fs.eventLog.maxLineLength` now also applies to the end-event reparse, so over-long lines there are skipped with a warning instead of being materialized.
    - Replay log messages on this path name the byte offset that their line numbers are relative to.
    - A line that starts exactly at the reparse offset is no longer discarded, so an end event there no longer forces a reparse of the whole log.
    - If the decompressed stream ends before the reparse offset, the scan no longer spins forever in the skip loop; the replay sees an empty input (previously, reaching EOF exactly at the offset failed with `NoSuchElementException` from `source.next()`).
    
    ### How was this patch tested?
    
    New tests in `FsHistoryProviderSuite`:
    - `end event reparse when the skip offset lands inside a multi-byte character`: the reparse offset is set to the third byte of a 3-byte character, so the line boundary scan starts on a continuation byte. Before this PR, the application is missing from the listing.
    - `end event reparse keeps a line that starts exactly at the skip offset`: the offset is the first byte of the end event, and a malformed line before it makes the whole-log fallback reparse fail, so the app is listed as completed only if the tail reparse finds the end event.
    - `end event reparse stops at the end of a stream shorter than the skip target`: a tiny lz4 log whose compressed file is larger than its content, with a 1-byte chunk; the scan must finish (it runs with a timeout).
    - `end event reparse applies the line length limit and reports relative line numbers` (uncompressed and lz4): an over-long line in the tail is skipped under a 1k limit, and the warning names the relative line number and the offset in the uncompressed stream.
    - `end event reparse reports JSON errors with the skip offset`: a malformed end event in the tail is reported with the relative line number and the offset, the path field stays the plain file name, and the application is not listed.
    - `skipToLineStart aligns to the first line starting at or after the target`: targets inside a line, at its `\n`, exactly at a line start, past EOF and non-positive; on each byte of a 3-byte character; CRLF; no trailing newline; a partial line longer than the read buffer; and a stream whose `skip` never advances.
    
    New test in `ReplayListenerSuite`:
    - `Replay from an offset reports where line numbers start`: `replayFromOffset` with a positive offset labels the over-long line warning, `Malformed line #N` and the truncated-file warning, and offset 0 leaves the diagnostics unchanged.
    
    `FsHistoryProviderSuite` (RocksDB backend), `ReplayListenerSuite`, `core/scalastyle` and `core/Test/scalastyle` pass on master.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59327 from viirya/SPARK-60109.
    
    Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
    (cherry picked from commit 003652f2ceb6a08d65168806fd1198c1bd4ee8da)
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
 .../spark/deploy/history/FsHistoryProvider.scala   |  74 ++++++--
 .../apache/spark/scheduler/ReplayListenerBus.scala |  51 ++++--
 .../deploy/history/FsHistoryProviderSuite.scala    | 191 +++++++++++++++++++++
 .../spark/scheduler/ReplayListenerSuite.scala      |  24 +++
 4 files changed, 314 insertions(+), 26 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index a379fba5eb88..64b62380195d 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -17,14 +17,13 @@
 
 package org.apache.spark.deploy.history
 
-import java.io.{File, FileNotFoundException, IOException}
+import java.io.{ByteArrayInputStream, File, FileNotFoundException, InputStream, IOException, SequenceInputStream}
 import java.lang.{Long => JLong}
 import java.util.{Date, NoSuchElementException, ServiceLoader}
 import java.util.concurrent.{ConcurrentHashMap, ExecutorService, TimeUnit}
 import java.util.zip.ZipOutputStream
 
 import scala.collection.mutable
-import scala.io.{Codec, Source}
 import scala.jdk.CollectionConverters._
 import scala.util.control.NonFatal
 import scala.xml.Node
@@ -1094,21 +1093,11 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
         if (target > 0) {
           logInfo(log"Looking for end event; skipping ${MDC(NUM_BYTES, target)} bytes" +
             log" from ${MDC(PATH, logPath)}...")
-          var skipped = 0L
-          while (skipped < target) {
-            skipped += in.skip(target - skipped)
-          }
-        }
-
-        val source = Source.fromInputStream(in)(Codec.UTF8).getLines()
-
-        // Because skipping may leave the stream in the middle of a line, read the next line
-        // before replaying.
-        if (target > 0) {
-          source.next()
         }
+        val (tail, startOffset) = skipToLineStart(in, target)
 
-        bus.replay(source, lastFile.getPath.toString, !appCompleted, eventsFilter)
+        bus.replayFromOffset(tail, lastFile.getPath.toString, startOffset, !appCompleted,
+          eventsFilter)
       }
     }
 
@@ -1741,6 +1730,61 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
 
 private[spark] object FsHistoryProvider {
 
+  /**
+   * Returns a stream that starts at the first line beginning at or after byte `target` of
+   * `in`, together with that line's byte offset. If `in` ends first, the returned stream is at
+   * EOF and the offset is the length of `in`. A non-positive `target` returns `in` unchanged
+   * with offset 0. Otherwise `in` is consumed and must not be read afterwards: bytes read past
+   * the line boundary are prepended to the returned stream, not pushed back into `in`.
+   *
+   * Since the skip may land anywhere, including inside a multi-byte UTF-8 character, the line
+   * boundary is found on raw bytes before any decoding: '\n' never occurs inside a multi-byte
+   * UTF-8 sequence.
+   */
+  private[history] def skipToLineStart(in: InputStream, target: Long): (InputStream, Long) = {
+    if (target <= 0) {
+      (in, 0L)
+    } else {
+      // Stop one byte early, so that a line starting exactly at `target` is kept.
+      var position = 0L
+      var eof = false
+      while (!eof && position < target - 1) {
+        val skipped = in.skip(target - 1 - position)
+        if (skipped > 0) {
+          position += skipped
+        } else if (in.read() == -1) {
+          // skip() may return 0 at EOF, e.g. when `target` comes from a compressed length that
+          // exceeds the decompressed one.
+          eof = true
+        } else {
+          position += 1
+        }
+      }
+
+      var result: InputStream = in
+      val buffer = new Array[Byte](8192)
+      var found = false
+      while (!eof && !found) {
+        val n = in.read(buffer)
+        if (n == -1) {
+          eof = true
+        } else {
+          var i = 0
+          while (i < n && buffer(i) != '\n') {
+            i += 1
+          }
+          position += math.min(i + 1, n)
+          if (i < n) {
+            found = true
+            val rest = new ByteArrayInputStream(buffer, i + 1, n - i - 1)
+            result = new SequenceInputStream(rest, in)
+          }
+        }
+      }
+      (result, position)
+    }
+  }
+
   private val APPL_START_EVENT_PREFIX = "{\"Event\":\"SparkListenerApplicationStart\""
 
   private val APPL_END_EVENT_PREFIX = "{\"Event\":\"SparkListenerApplicationEnd\""
diff --git a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
index 69720567d34d..24b7e2a9d3d6 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
@@ -26,7 +26,7 @@ import com.fasterxml.jackson.core.{JsonParseException, JsonProcessingException}
 import com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException
 
 import org.apache.spark.SparkConf
-import org.apache.spark.internal.Logging
+import org.apache.spark.internal.{Logging, MessageWithContext}
 import org.apache.spark.internal.LogKeys._
 import org.apache.spark.internal.config.History
 import org.apache.spark.scheduler.ReplayListenerBus._
@@ -72,8 +72,23 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
-    val lines = boundedLines(logData, sourceName)
-    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
+    replayFromOffset(logData, sourceName, 0L, maybeTruncated, eventsFilter)
+  }
+
+  /**
+   * Variant of [[replay()]] for a stream positioned at the start of a line that begins at
+   * `startOffset` bytes into the (uncompressed) source. A positive `startOffset` is added to
+   * the diagnostics so that the reported line numbers, which count from the start of the
+   * stream, can be located in the source.
+   */
+  private[spark] def replayFromOffset(
+      logData: InputStream,
+      sourceName: String,
+      startOffset: Long,
+      maybeTruncated: Boolean,
+      eventsFilter: ReplayEventsFilter): Boolean = {
+    val lines = boundedLines(logData, sourceName, startOffset)
+    replayEntries(lines, sourceName, startOffset, maybeTruncated, eventsFilter)
   }
 
   /**
@@ -84,7 +99,8 @@ private[spark] class ReplayListenerBus(
    */
   private def boundedLines(
       logData: InputStream,
-      sourceName: String): Iterator[(String, Int)] = {
+      sourceName: String,
+      startOffset: Long): Iterator[(String, Int)] = {
     // Fail on malformed input like Source.getLines() does instead of replacing it.
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)
@@ -146,10 +162,12 @@ private[spark] class ReplayListenerBus(
             if (!warned) {
               logWarning(log"Skipped event log lines longer than " +
                 log"${MDC(MAX_SIZE, effectiveMaxLineLength)} bytes in " +
-                log"${MDC(FILE_NAME, sourceName)}; first skipped line: ${MDC(LINE_NUM, index + 1)}")
+                log"${MDC(FILE_NAME, sourceName)}; first skipped line: " +
+                log"${MDC(LINE_NUM, index + 1)}" + lineOrigin(startOffset))
               warned = true
             }
-            logDebug(s"Skipped event log line ${index + 1} in $sourceName")
+            logDebug(s"Skipped event log line ${index + 1} in $sourceName" +
+              lineOrigin(startOffset).message)
             fetchLine()
           } else {
             (maybeLine.get, index)
@@ -168,12 +186,13 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
-    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines.zipWithIndex, sourceName, 0L, maybeTruncated, eventsFilter)
   }
 
   private def replayEntries(
       lines: Iterator[(String, Int)],
       sourceName: String,
+      startOffset: Long,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
     var currentLine: String = null
@@ -217,8 +236,8 @@ private[spark] class ReplayListenerBus(
               throw jpe
             } else {
               logWarning(log"Got JsonParseException from log file ${MDC(FILE_NAME, sourceName)}" +
-                log" at line ${MDC(LINE_NUM, lineNumber)}, " +
-                log"the file might not have finished writing cleanly.")
+                log" at line ${MDC(LINE_NUM, lineNumber)}" + lineOrigin(startOffset) +
+                log", the file might not have finished writing cleanly.")
             }
         }
       }
@@ -230,7 +249,7 @@ private[spark] class ReplayListenerBus(
       case _: EOFException if maybeTruncated => false
       case jpe: JsonProcessingException =>
         logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)} " +
-          log"at line ${MDC(LINE_NUM, lineNumber)}")
+          log"at line ${MDC(LINE_NUM, lineNumber)}" + lineOrigin(startOffset))
         throw jpe
       case ioe: IOException =>
         throw ioe
@@ -238,12 +257,22 @@ private[spark] class ReplayListenerBus(
         logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)}", e)
         val prefix = Option(currentLine).map(_.take(1024)).orNull
         val length = Option(currentLine).fold(0)(_.length)
-        logError(log"Malformed line #${MDC(LINE_NUM, lineNumber)}: ${MDC(LINE, prefix)} " +
+        logError(log"Malformed line #${MDC(LINE_NUM, lineNumber)}" + lineOrigin(startOffset) +
+          log": ${MDC(LINE, prefix)} " +
           log"(line length: ${MDC(SIZE, length)} characters; showing at most 1024)\n")
         false
     }
   }
 
+  /** Describes where line numbers start when the replay does not start at the source start. */
+  private def lineOrigin(startOffset: Long): MessageWithContext = {
+    if (startOffset > 0) {
+      log" (lines counted from uncompressed byte offset ${MDC(OFFSET, startOffset)})"
+    } else {
+      log""
+    }
+  }
+
   override protected def isIgnorableException(e: Throwable): Boolean = {
     e.isInstanceOf[HaltReplayException]
   }
diff --git a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
index 829010179bda..6589607e6e93 100644
--- a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
@@ -1169,6 +1169,154 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite with Matchers with P
     }
   }
 
+  test("end event reparse when the skip offset lands inside a multi-byte character") {
+    val log = newLogFile("end-event-utf8", None, inProgress = false)
+    val bytes = writeRawLog(log, endEventTestLines("end-event-utf8",
+      jobStartLine(1, 0x4e2d.toChar.toString * 200)))
+    // The job description is written as raw UTF-8. Target the third byte of a 3-byte character,
+    // so that the line boundary scan, which starts one byte before the target, starts on a
+    // continuation byte.
+    val leadByte = bytes.indexWhere(b => (b & 0xF0) == 0xE0)
+    assert(leadByte > 0)
+    val conf = createTestConf().set(END_EVENT_REPARSE_CHUNK_SIZE, (bytes.length - leadByte - 2L))
+    updateAndCheck(new FsHistoryProvider(conf)) { list =>
+      assert(list.size === 1)
+      assert(list(0).attempts.head.completed)
+    }
+  }
+
+  test("end event reparse keeps a line that starts exactly at the skip offset") {
+    val log = newLogFile("end-event-boundary", None, inProgress = false)
+    // The whole-log fallback reparse fails on this malformed line, so the application can only
+    // be listed as completed if the tail reparse finds the end event.
+    val malformed = "{\"Event\":\"SparkListenerApplicationEnd\",\"Timestamp\":"
+    val lines = endEventTestLines("end-event-boundary", malformed)
+    writeRawLog(log, lines)
+    // The application end event is the last line: skip exactly to its first byte.
+    val conf = createTestConf()
+      .set(END_EVENT_REPARSE_CHUNK_SIZE, lineBytes(lines.last).toLong)
+    updateAndCheck(new FsHistoryProvider(conf)) { list =>
+      assert(list.size === 1)
+      assert(list(0).attempts.head.completed)
+    }
+  }
+
+  test("end event reparse stops at the end of a stream shorter than the skip target") {
+    val log = newLogFile("end-event-short", None, inProgress = false, Some(CompressionCodec.LZ4))
+    val codec = CompressionCodec.createCodec(new SparkConf(), CompressionCodec.LZ4)
+    val bytes = writeRawLog(log, Seq(JsonProtocol.sparkEventToJsonString(
+      SparkListenerApplicationStart("a", Some("b"), 1L, "c", None))), Some(codec))
+    // The lz4 framing makes this tiny log larger than its content, so with a 1-byte chunk the
+    // skip target is past the end of the decompressed stream: skip() returns 0 there.
+    assert(log.length() - 2 > bytes.length)
+    val provider = new FsHistoryProvider(
+      createTestConf().set(END_EVENT_REPARSE_CHUNK_SIZE, 1L))
+    // Run the scan in a separate thread so that a skip loop that never ends fails the test.
+    val scan = new Thread(() => provider.checkForLogs())
+    scan.setDaemon(true)
+    scan.start()
+    scan.join(TimeUnit.MINUTES.toMillis(1))
+    assert(!scan.isAlive, "the scan did not finish")
+    assert(provider.getListing().toSeq.map(_.id) === Seq("b"))
+  }
+
+  Seq(None, Some(CompressionCodec.LZ4)).foreach { codecName =>
+    test("end event reparse applies the line length limit and reports relative line numbers " +
+        s"(codec = ${codecName.getOrElse("none")})") {
+      val log = newLogFile("end-event-limit", None, inProgress = false, codecName)
+      val tail = Seq(jobStartLine(2, "x" * 2048), jobStartLine(3, "y"))
+      val lines = endEventTestLines("end-event-limit", tail: _*)
+      val codec = codecName.map(CompressionCodec.createCodec(new SparkConf(), _))
+      writeRawLog(log, lines, codec)
+      // The skip runs on the uncompressed stream while the chunk size is subtracted from the
+      // file length. Skip into the middle of the first line: replay then starts at the second
+      // line, and the over-long line is at relative line `lines.indexOf(tail.head)`.
+      val target = 5L
+      val conf = createTestConf()
+        .set(END_EVENT_REPARSE_CHUNK_SIZE, log.length() - target)
+        .set(EVENT_LOG_MAX_LINE_LENGTH.key, "1k")
+      val appender = new LogAppender("end event reparse")
+      withLogAppender(appender) {
+        updateAndCheck(new FsHistoryProvider(conf)) { list =>
+          assert(list.size === 1)
+          assert(list(0).attempts.head.completed)
+        }
+      }
+      val offset = lineBytes(lines.head)
+      val messages = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+      assert(messages.exists(m => m.contains("Skipped event log lines longer than 1024 bytes") &&
+        m.contains(s"${log.getName}; first skipped line: ${lines.indexOf(tail.head)} " +
+          s"(lines counted from uncompressed byte offset $offset)")), messages)
+    }
+  }
+
+  test("end event reparse reports JSON errors with the skip offset") {
+    val log = newLogFile("end-event-malformed", None, inProgress = false)
+    val malformed = "{\"Event\":\"SparkListenerApplicationEnd\",\"Timestamp\":"
+    val tail = Seq(jobStartLine(2, "x"), malformed, jobStartLine(3, "y"))
+    val lines = endEventTestLines("end-event-malformed", tail: _*)
+    val bytes = writeRawLog(log, lines)
+    val offset = lines.takeWhile(_ != tail.head).map(lineBytes).sum
+    val conf = createTestConf().set(END_EVENT_REPARSE_CHUNK_SIZE, bytes.length - offset + 10L)
+    val appender = new LogAppender("end event reparse")
+    withLogAppender(appender) {
+      // A completed log fails on a malformed line that is not the last one, so the application
+      // is not listed.
+      updateAndCheck(new FsHistoryProvider(conf)) { list =>
+        assert(list.isEmpty)
+      }
+    }
+    val messages = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+    assert(messages.exists(_.contains(s"${log.getName} at line 2 " +
+      s"(lines counted from uncompressed byte offset $offset)")), messages)
+    assert(!messages.exists(_.contains(s"${log.getName} (lines")), messages)
+  }
+
+  test("skipToLineStart aligns to the first line starting at or after the target") {
+    def alignStream(in: InputStream, target: Long): (String, Long) = {
+      val (rest, offset) = FsHistoryProvider.skipToLineStart(in, target)
+      (new String(rest.readAllBytes(), StandardCharsets.UTF_8), offset)
+    }
+    def align(data: String, target: Long): (String, Long) = {
+      alignStream(new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8)), target)
+    }
+    // Inside a line, at its '\n', and exactly at the start of the next line.
+    assert(align("abc\ndef\n", 1) === (("def\n", 4L)))
+    assert(align("abc\ndef\n", 3) === (("def\n", 4L)))
+    assert(align("abc\ndef\n", 4) === (("def\n", 4L)))
+    assert(align("abc\ndef\n", 5) === (("", 8L)))
+    // CRLF line endings, and a target at the '\r'.
+    assert(align("ab\r\ncd", 2) === (("cd", 4L)))
+    assert(align("ab\r\ncd", 4) === (("cd", 4L)))
+    // No line starts at or after the target, or the target is beyond the end of the stream.
+    assert(align("abc", 1) === (("", 3L)))
+    assert(align("abc\n", 10) === (("", 4L)))
+    // Targets on each byte of a 3-byte character, and on the '\n' after it.
+    val cjk = "a" + 0x4e2d.toChar + "\nb"
+    (1 to 5).foreach { target =>
+      assert(align(cjk, target) === (("b", 5L)), s"target = $target")
+    }
+    // A non-positive target leaves the stream unchanged.
+    assert(align("abc\ndef", 0) === (("abc\ndef", 0L)))
+    assert(align("abc\ndef", -1) === (("abc\ndef", 0L)))
+    // A partial line longer than the read buffer.
+    val long = "x" * 20000 + "\nnext"
+    assert(align(long, 1) === (("next", 20001L)))
+    // More than a read buffer of varied content after the boundary, so the rest of the
+    // stream must follow the put-back bytes in order.
+    val longTail = (1 to 3000).mkString(",")
+    assert(longTail.length > 8192)
+    assert(align("abc\n" + longTail, 1) === ((longTail, 4L)))
+    // A stream whose skip() never advances, e.g. at the end of a decompressed stream.
+    def noSkip(data: String): InputStream = {
+      new FilterInputStream(new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8))) {
+        override def skip(n: Long): Long = 0L
+      }
+    }
+    assert(alignStream(noSkip("abc\ndef"), 2) === (("def", 4L)))
+    assert(alignStream(noSkip("abc\n"), 100) === (("", 4L)))
+  }
+
   test("parse event logs with optimizations off") {
     val conf = createTestConf()
       .set(END_EVENT_REPARSE_CHUNK_SIZE, 0L)
@@ -1896,6 +2044,49 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite with Matchers with P
     }
   }
 
+  /** Writes the given lines as an event log and returns the uncompressed bytes. */
+  private def writeRawLog(
+      file: File,
+      lines: Seq[String],
+      codec: Option[CompressionCodec] = None): Array[Byte] = {
+    val bytes = lines.map(_ + "\n").mkString.getBytes(StandardCharsets.UTF_8)
+    Utils.tryWithResource(new FileOutputStream(file)) { out =>
+      Utils.tryWithResource(codec.map(_.compressedContinuousOutputStream(out)).getOrElse(out)) {
+        _.write(bytes)
+      }
+    }
+    bytes
+  }
+
+  /** UTF-8 length of a line including its '\n'. */
+  private def lineBytes(line: String): Int = line.getBytes(StandardCharsets.UTF_8).length + 1
+
+  private def jobStartLine(jobId: Int, description: String): String = {
+    val properties = new java.util.Properties()
+    properties.setProperty("spark.job.description", description)
+    JsonProtocol.sparkEventToJsonString(SparkListenerJobStart(jobId, 1L, Nil, properties))
+  }
+
+  /**
+   * Event log lines with enough information for a listing entry, then a padding line, the given
+   * tail lines and the application end event.
+   */
+  private def endEventTestLines(appId: String, tail: String*): Seq[String] = {
+    Seq(
+      SparkListenerLogStart(SPARK_VERSION),
+      SparkListenerApplicationStart(appId, Some(appId), 1L, "test", None),
+      SparkListenerEnvironmentUpdate(Map(
+        "Spark Properties" -> Seq.empty,
+        "Hadoop Properties" -> Seq.empty,
+        "JVM Information" -> Seq.empty,
+        "System Properties" -> Seq.empty,
+        "Metrics Properties" -> Seq.empty,
+        "Classpath Entries" -> Seq.empty))
+    ).map(JsonProtocol.sparkEventToJsonString) ++
+      Seq(jobStartLine(1, "padding")) ++ tail ++
+      Seq(JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(5L)))
+  }
+
   private def createEmptyFile(file: File) = {
     new FileOutputStream(file).close()
   }
diff --git a/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala b/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
index a2802f3a8f70..ad496b5540ff 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
@@ -405,6 +405,30 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(warnings.forall(_.contains("at line 3,")))
   }
 
+  test("Replay from an offset reports where line numbers start") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    def replay(data: String, startOffset: Long, maybeTruncated: Boolean): Seq[String] = {
+      val appender = new LogAppender
+      withLogAppender(appender) {
+        new ReplayListenerBus(1024).replayFromOffset(
+          new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8)), "offset",
+          startOffset, maybeTruncated, ReplayListenerBus.SELECT_ALL_FILTER)
+      }
+      appender.loggingEvents.map(_.getMessage.getFormattedMessage).toSeq
+    }
+    val origin = " (lines counted from uncompressed byte offset 42)"
+    val malformed = replay(end + "\n" + "x" * 2048 + "\n{}\n", 42L, maybeTruncated = false)
+    assert(malformed.exists(_.contains("first skipped line: 2" + origin)), malformed)
+    assert(malformed.exists(_.startsWith("Malformed line #3" + origin + ": {}")), malformed)
+    val truncated = replay(end + "\n{\"Event\":", 42L, maybeTruncated = true)
+    assert(truncated.exists(_.contains("offset at line 2" + origin + ", the file might not")),
+      truncated)
+    // Without an offset, the diagnostics are unchanged.
+    val unlabeled = replay(end + "\n{}\n", 0L, maybeTruncated = false)
+    assert(unlabeled.exists(_.startsWith("Malformed line #2: {}")), unlabeled)
+    assert(!unlabeled.exists(_.contains("lines counted from")), unlabeled)
+  }
+
   test("SPARK-59804: Replay still rejects malformed UTF-8 in skipped lines") {
     val bytes = Array.fill[Byte](20 * 1024)('x'.toByte) ++ Array(0xff.toByte, '\n'.toByte)
     val bus = new ReplayListenerBus(1024)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org