You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/09/27 00:14:14 UTC

[PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

viirya opened a new pull request, #59074:
URL: https://github.com/apache/spark/pull/59074

   ### What changes were proposed in this pull request?
   
   Fix two regressions introduced by SPARK-59407 (#58700) in `ReplayListenerBus`:
   
   - Measure `spark.history.fs.eventLog.maxLineLength` in UTF-8 content bytes, excluding line endings, and report the warning in bytes.
   - Preserve physical line indices when oversized lines are skipped so subsequent parsing diagnostics identify the correct file line.
   
   Clarify the configuration documentation and add regression coverage. The reader continues to drain oversized lines with bounded buffering and rejects malformed UTF-8 as before. The iterator-based replay API retains its existing behavior.
   
   JIRA: https://issues.apache.org/jira/browse/SPARK-59804
   
   ### Why are the changes needed?
   
   The existing reader compares `StringBuilder.length() * 2` with a byte-valued setting. With an 8 KiB limit, a valid ASCII application-start event with a 6,000-character application name is incorrectly skipped. Multibyte UTF-8 characters also produce inconsistent boundaries, and CRLF can reject content exactly at the limit.
   
   The reader also drops oversized lines before `zipWithIndex`. A valid event on line 1, oversized lines 2 and 3, and an invalid event on line 4 therefore report `Malformed line #2` instead of `Malformed line #4`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Events at or below the configured UTF-8 byte limit are replayed, LF/CRLF endings do not consume the content limit, and parsing diagnostics retain physical file line numbers after skipped lines. These correct regressions introduced by SPARK-59407 on the development and maintenance branches.
   
   The setting remains a line-content limit rather than an exact JVM heap budget; memory use is bounded in proportion to the configured limit.
   
   ### How was this patch tested?
   
   Added tests for approximately 6 KiB of ASCII content under an 8 KiB limit; one- through four-byte UTF-8 characters; exact and over-limit boundaries; decoder-buffer crossings; LF, CRLF, and unterminated final lines; physical line numbers after skipped lines; filtered iterators; truncated logs; and malformed UTF-8 in oversized lines.
   
   Three new regression tests fail against the original implementation when compiled and run in isolation with the existing dependency classpath.
   
   Validation with SBT:
   
   ```
   build/sbt 'core/testOnly *ReplayListenerSuite'
   build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle'
   ```
   
   All 11 suite tests passed, including compressed-log and end-to-end replay coverage. Production and test scalastyle checks passed. After the localized non-ASCII test-data exemption was added, the affected UTF-8 boundary test and test scalastyle were rerun and passed.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #59074: [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay
URL: https://github.com/apache/spark/pull/59074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4141946296


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   If not `3.5.10`, what version string we should use in these branches? `4.3.0` as the doc said that it is introduced in 4.3.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5916376640

   +1 for @viirya 's decision ("I prefer to focus on the bug in this PR and leave the config version unchanged").


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5917273845

   Thanks @dongjoon-hyun @HyukjinKwon @uros-b @HeartSaVioR !
   
   Only branch-3.5 requires a manual backport. I will create a backport PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4119010915


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -242,4 +248,37 @@ private[spark] object ReplayListenerBus {
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()
+    private var byteLength = 0L
+    private val bufferLimit = maxLineLength.toLong + 1

Review Comment:
   This retains up to `maxLineLength + 1` chars, but nothing caps that against the `StringBuilder` implementation limits, so larger configured limits now fail with `OutOfMemoryError`, regardless of `-Xmx`, where the current code skips the line:
   
   - **Latin-1 to UTF-16 inflation (limits >= 603,979,775 bytes, i.e. from 576m).** Once more than 603,979,774 Latin-1 chars are retained, the builder's capacity is 1,207,959,550, and the first char >= U+0100 makes `inflate()` fail with `UTF16 String size is 1207959550, should be less than 1073741823`. This also hits lines within the limit: with `600m`, 620,000,000 `x` followed by U+0100 (620,000,002 bytes) throws, while the current code skips it. The current code needs 1152m or more for this. It is the [576m, 1152m) case from the first round, which the new default avoids only for the default.
   - **UTF-16 maximum length (limits >= 1,073,741,823 bytes, about 1g).** A UTF-16 builder cannot exceed 1,073,741,822 chars, so e.g. `€` followed by about 1.18 billion `x` at `1g` throws `Requested array size exceeds VM limit`, while the current code skips it.
   - `<= 0` and values above `Int.MaxValue` fail the same way as the current code, but the new doc now calls 2147483647 bytes the "maximum supported limit".
   
   On the listing path, the `Error` passes `mergeApplicationListing`'s `case e: Exception` and is swallowed by the `Future` of `pool.submit` in `submitLogProcessTask`, so the app is silently missing from the listing. `256m` and an explicit `512m` are safe.
   
   Could we clamp the effective limit in `maxLineLength(conf)` (and in the constructor if needed), e.g. to `512m` (any value up to 603,979,773 keeps the capacity at or below 603,979,774), and document that as the maximum? The clamp needs to apply to the limit itself, not only to `bufferLimit`, so that lines within the limit are not silently truncated. I reproduced these with a verbatim copy of the reader on JDK 21 (up to `-Xmx8g`).



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,18 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 2147483647, use the maximum supported limit of " +
+        "2147483647 bytes. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .bytesConf(ByteUnit.BYTE)
-      .createWithDefaultString("512m")
+      .createWithDefaultString("256m")

Review Comment:
   A reminder about the backports from the first round: this changes the unit and the default of a key that branch-4.x, branch-4.3, branch-4.2, branch-4.1, branch-4.0 and branch-3.5 already have (all still `512m` with the `chars * 2` check, and not in any release yet).
   
   - [v4.3.0-rc1](https://github.com/apache/spark/tree/v4.3.0-rc1) (2026-08-31) does not contain SPARK-59407, but `branch-4.3` does, so this needs to land on `branch-4.3` before the next 4.3.0 RC. Otherwise 4.3.0 ships the old semantics (e.g. `8k` means about 4096 chars and `<= 0` means no limit), and changing it afterwards becomes a behavior change of a released config.
   - The diff cherry-picks cleanly to branch-4.x and branch-4.3, and branch-4.2/4.1/4.0 conflict only in the scaladoc (`byes`/`bytes`). On branch-3.5, the new `case jpe: JsonProcessingException` block with `log"..."`/`MDC` merges without a conflict but does not compile, so it needs a separate PR like #59058.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +66,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(
+      logData: InputStream,
+      sourceName: String): Iterator[(String, Int)] = {
     // Fail on malformed input like Source.getLines() does instead of replacing it.
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)

Review Comment:
   Pre-existing and preserved by this PR (`Source.getLines()` with `Codec.UTF8` also fails on malformed input), but related to the truncation handling: if the last, unterminated line of an in-progress or crashed app's log ends inside a multi-byte UTF-8 sequence, this decoder throws `MalformedInputException` from `reader.read()` at EOF. That happens in `lineEntries.hasNext`, outside the inner `try`, so it never reaches the `maybeTruncated` tolerance for `JsonParseException`; it is rethrown as an `IOException`, and the UI rebuild fails after its retry (permanently for a crashed app). The same log cut at an ASCII boundary replays fine.
   
   Compressed in-progress logs expose lengths at compression block boundaries (128 KiB blocks for zstd, 32 KiB for lz4) rather than at code point boundaries, so with CJK-heavy content this happened in 3/80 (zstd) and 5/80 (lz4) simulated cuts; uncompressed local files were always cut at code point boundaries in my runs. A follow-up could treat a `MalformedInputException` raised after the underlying stream hit EOF as a truncated last line when `maybeTruncated` is set.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,149 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == Int.MaxValue)
+    }
+  }
+
+  test("Replay preserves physical line numbers after skipping long lines") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      (end + "\n" + "x" * 2048 + "\n" + "x" * 2048 + "\n{}\n")
+        .getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(!bus.replay(input, "line-numbers"))
+    }
+    assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Malformed line #4: {}")))
+  }
+
+  test("Replay logs physical line numbers before rethrowing JSON errors") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val mappingError =
+      """{"Event":"org.apache.spark.util.TestListenerEvent","foo":"x","bar":[]}"""
+    for ((malformed, errorClass) <- Seq(
+        ("{bad", classOf[JsonParseException]),
+        (mappingError, classOf[MismatchedInputException]))) {
+      val appender = new LogAppender
+      val bus = new ReplayListenerBus(1024)
+      val input = new ByteArrayInputStream(
+        Seq(end, "x" * 2048, "x" * 2048, malformed, end).mkString("\n")
+          .getBytes(StandardCharsets.UTF_8))
+      withLogAppender(appender) {
+        val error = intercept[JsonProcessingException] {
+          bus.replay(input, "json-errors", maybeTruncated = true)
+        }
+        assert(errorClass.isInstance(error))
+      }
+      assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+        .contains("Exception parsing Spark event log: json-errors at line 4")))
+    }
+  }
+
+  test("Replay propagates stream IO errors without reporting a stale parse line") {
+    val error = new IOException("read failure")
+    val input = new InputStream {
+      override def read(): Int = throw error
+    }
+    val appender = new LogAppender
+    withLogAppender(appender) {
+      assert(intercept[IOException] {
+        new ReplayListenerBus().replay(input, "read-error")
+      } eq error)
+    }
+    assert(!appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Exception parsing Spark event log")))
+  }
+
+  test("Replay preserves line numbers for truncated logs and filtered iterators") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val truncated = "{\"Event\":"
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      ("x" * 2048 + "\n" + end + "\n" + truncated).getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(bus.replay(input, "truncated", maybeTruncated = true,
+        eventsFilter = _ != end))
+      assert(bus.replay(Iterator(end, end, truncated), "iterator", maybeTruncated = true,
+        eventsFilter = _ != end))
+    }
+    val warnings = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+      .filter(_.contains("Got JsonParseException"))
+    assert(warnings.size == 2)
+    assert(warnings.forall(_.contains("at line 3,")))
+  }
+
+  test("Replay still rejects malformed UTF-8 in skipped lines") {
+    val bytes = Array.fill[Byte](20 * 1024)('x'.toByte) ++ Array(0xff.toByte, '\n'.toByte)
+    val bus = new ReplayListenerBus(1024)
+    intercept[java.nio.charset.MalformedInputException] {
+      bus.replay(new ByteArrayInputStream(bytes), "invalid-utf8")
+    }
+  }
+
   /**
    * Test replaying compressed spark history file that internally throws an EOFException.  To

Review Comment:
   About the "existing compressed/end-to-end replay tests" in the PR description: they don't really check the replayed events (pre-existing, not from this PR).
   
   - `testApplicationReplay`, used by "End-to-end replay" and "End-to-end replay with compression", compares each event with itself, `JsonProtocolSuite.assertEquals(e1, e1)` (L408). SPARK-28770 changed it from `(e1, e2)`, which looks like a typo. It also `zip`s without a size check and ignores the result of `replay` (L396), so it passes even if nothing is replayed.
   - This test only checks the event count (L300) and also ignores the result of `replay` (L298).
   
   So nothing checks the content of a multi-buffer LZ4/Snappy/ZSTD log read through the new reader. Could we assert the result, compare `originalEvents.size` with `replayedEvents.size` and use `assertEquals(e1, e2)` (and pick the current app's log instead of the latest file in the shared directory), here or in a small test-only follow-up? Otherwise, please adjust the claim in the description.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -202,6 +205,10 @@ private[spark] class ReplayListenerBus(
         // Just stop replay.
         false
       case _: EOFException if maybeTruncated => false
+      case jpe: JsonProcessingException =>
+        logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)} " +
+          log"at line ${MDC(LINE_NUM, lineNumber)}", jpe)

Review Comment:
   Thanks for adding the location. Passing `jpe` here duplicates the stack trace, though, since every caller already logs or retries the same `IOException`:
   
   - UI loads: `createHybridStore`, `createDiskStore` and `createInMemoryStore` (FsHistoryProvider L1598, L1672, L1701) retry once on `IOException`, so one malformed line now adds 2 ERROR stack traces per request (4 with the hybrid store), where the current code logs none at ERROR (only the INFO "trying again"). Failed loads are not cached, so this repeats on every refresh.
   - Listing (L1032) and compaction (L1224) log the same exception again at ERROR.
   
   Could we drop the throwable and keep only the source and line (optionally with `jpe.getOriginalMessage`)? The new test only checks the message, so it would still pass.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +145,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)

Review Comment:
   Two notes for the tail-reparse follow-up (FsHistoryProvider L1104-L1127), which calls this overload:
   
   1. There, `lines` starts after the byte skip and the dropped partial line, so the new ERROR at L208-L211 prints the full path with a skip-relative number. For example, a truncated `SparkListenerApplicationEnd` line at physical line 12,004 of a completed app is reported as `at line 3959`. Before this PR, JSON errors on this path had no bus-level line number. Please cover it in the follow-up, together with the existing `Malformed line #N` and the truncation WARN, which are also skip-relative there, or label the source on that path, e.g. `s"$path (lines counted after skipping $target bytes)"`.
   2. Pre-existing: if `target` lands inside a multi-byte UTF-8 sequence (e.g. CJK SQL text or job descriptions, which Jackson writes as raw UTF-8), `source.next()` (L1124) throws `MalformedInputException` before `replay` is called. `mergeApplicationListing` only logs it, and since the placeholder `LogInfo` was already written with the same file size, `shouldReloadLog` never retries it, so the completed app stays missing from the listing (and `target` is deterministic, so a restart fails the same way). Replaying through `boundedLines` after the skip, as planned, would throw the same way because of its strict decoder, so the follow-up should drop the bytes up to the first `\n` before decoding. I reproduced it at the unit level, not yet on a real SHS.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -226,13 +233,12 @@ private[spark] class HaltReplayException extends RuntimeException
 private[spark] object ReplayListenerBus {
 
   /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
+   * Default UTF-8 line-length cap, matching spark.history.fs.eventLog.maxLineLength.
+   * Keep the maximum buffered character count close to the original 512 MiB / 2 cap.
    */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  val DEFAULT_MAX_LINE_LENGTH: Int = 256 * 1024 * 1024

Review Comment:
   nit: This duplicates the `256m` default of `History.EVENT_LOG_MAX_LINE_LENGTH`, and this PR had to change both. Deriving it, e.g. `History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue.get.toInt` (like `ChunkedByteBuffer` does with `BUFFER_WRITE_CHUNK_SIZE`; the API is the same down to branch-3.5), keeps them in sync. Also, "the original 512 MiB / 2 cap" refers to the unreleased code that this PR replaces, so it will be hard to interpret after the merge. Could the comment describe the current bound instead (a skipped line retains at most this many chars plus a possible trailing CR while it is drained) and leave the history to the PR description?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4131003225


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   Note that I found the backport PRs of [SPARK-59407](https://issues.apache.org/jira/browse/SPARK-59407) fills this `.version` field of this config with the version number in each branch, i.e., `.version("4.1.4")` on branch-4.1 backport, `.version("3.5.10")` on branch-3.5 backport.
   
   It makes this config look a bit strange as on each version, this config has different version value on different branch. But what `.version` specifies should be the version that the config is added.
   
   It seems because this is an improvement ticket and adds a new config, but it backports to many previous branches so the config in earlier versions looks strange. 
   
   I'm not sure how to deal with it when to backport this fix to these branches. Just backport this so all branches have "4.3.0" in `.version` of this config? Or leave the strange `version` unchanged? 🤔 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4131009091


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   @dongjoon-hyun @HyukjinKwon What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5917259929

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/719947a5652f83a80492db50a6f1e5fe14d9e6b8
   - merged into branch-4.x https://github.com/apache/spark/commit/52c7fa15b720bb816d395301f7e5fe2f9404943d
   - merged into branch-4.3 https://github.com/apache/spark/commit/cd32ffa957d1e2ca46786348fc9083e09456084b
   - merged into branch-4.2 https://github.com/apache/spark/commit/eabf40f3730da66c38bbfd08bed99b4b094279e1
   - merged into branch-4.1 https://github.com/apache/spark/commit/b4799fd39d22f2b3c5f2b247c5db49abd28c0434
   - merged into branch-4.0 https://github.com/apache/spark/commit/d99a2d9766cbc91518f2fb6b9d6bcf8e231850e9
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5906311937

   @HyukjinKwon @dongjoon-hyun Do you have any more comments except for the `version` of the config? Since this is set as a blocker, we should move this forward soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4126668723


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +69,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(
+      logData: InputStream,
+      sourceName: String): Iterator[(String, Int)] = {
     // Fail on malformed input like Source.getLines() does instead of replacing it.
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)
       .onUnmappableCharacter(CodingErrorAction.REPORT)
     val reader = new BufferedReader(new InputStreamReader(logData, decoder))

Review Comment:
   We will evaluate raw-byte splitting in the reader follow-up, preserving strict validation of skipped content and covering tail alignment and incomplete UTF-8 at EOF. I have not independently reproduced the benchmark numbers.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex

Review Comment:
   I would keep the current iterator in this fix to limit the backport diff. We can revisit the iterator structure with the reader follow-up.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)

Review Comment:
   Agreed to evaluate buffer reuse in the reader follow-up, including a retention cap after unusually large lines. I have not independently reproduced the allocation or throughput measurements.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {

Review Comment:
   Added the SPARK-59804 prefix to the new test names.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,19 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 536870912, use the maximum supported limit of " +

Review Comment:
   The maximum now lives in History, and both the config documentation and ReplayListenerBus derive it from that constant. monitoring.md retains the corresponding documented value.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -35,15 +35,17 @@ import org.apache.spark.util.JsonProtocol
 /**
  * A SparkListenerBus that can be used to replay events from serialized event data.
  *
- * @param maxLineLength Maximum number of byes (1/2 chars) of a single event log line that will be
- *                      materialized during replay. Longer lines are drained, skipped and
- *                      logged, bounding the memory replay can use when an event log is
- *                      corrupt or unexpectedly large.
+ * @param maxLineLength Maximum UTF-8 byte length of a single event log line, excluding its line

Review Comment:
   The constructor scaladoc now describes the normalization of non-positive and above-maximum values to the 512 MiB limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4117451556


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {

Review Comment:
   Confirmed. I reproduced the OOM in a separate `-Xmx1g` JVM using a lazy stream containing U+0101 followed by 600,000,000 ASCII characters. The 512 MiB limit fails during builder growth; the revised implementation with a 256 MiB limit successfully drains and skips the same input.
   
   Changed both defaults to `256m` in ab7c85da8f6 to keep the maximum buffered character count close to the previous effective bound. The configuration docs now describe the additional heap cost of buffer growth, UTF-16 storage and JSON parsing. This does not guarantee that every accepted 256 MiB event fits in a 1 GiB heap.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true

Review Comment:
   Addressed in ab7c85da8f6. The reader now uses a `BoundedLineBuffer`, and the new test checks its retained character count after every append while feeding content well beyond the limit. It covers zero and small limits and allows only the one extra character reserved for a possible trailing CR.
   
   I also tested mutations that remove the append cap or double it. Both cause this test to fail, even though the oversized line would still be rejected at the end.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true
+              }
             }
             c = reader.read()

Review Comment:
   Agreed that this deserves a separate follow-up. We will address block-oriented reading and validate it with an SHS replay benchmark there. This PR keeps the existing read strategy and focuses on the byte-limit and diagnostic fixes; I have not independently rerun the throughput measurements quoted here.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true
+              }
             }
             c = reader.read()
           }
-          if (overLong) {
+          if (sb.length() > 0 && sb.charAt(sb.length() - 1) == '\r') {
+            sb.setLength(sb.length() - 1)
+            byteLength -= 1
+          }
+          if (overLong || byteLength > maxLineLength) {

Review Comment:
   Removed `overLong` in ab7c85da8f6. The bounded accumulator stops updating once the byte count exceeds the buffer allowance, and the final content-byte check determines whether to skip the line. This also avoids continuing to count the rest of an arbitrarily long discarded line.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {

Review Comment:
   Applied in ab7c85da8f6: each surrogate half contributes two bytes, with a comment explaining the strict decoder's valid-pair guarantee. The new raw UTF-8 tests exercise supplementary code points and limits inside those code points. A mutation counting three bytes per half fails those tests.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +170,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)

Review Comment:
   We will handle this in a separate follow-up. The tail-reparse reader and its offset/partial-line semantics are independent of the two issues addressed here. This PR leaves that path unchanged; the follow-up will address the bypass of the bounded reader and clarify line-number semantics after skipping into a file.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +66,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(

Review Comment:
   We will address compaction reader and retention consistency in a separate follow-up, including reproducing the possible loss of live-entity events. That needs its own retention tests and a decision about handling oversized start events across both passes. This PR does not change the compaction rewrite path.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +170,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)
+  }
+
+  private def replayEntries(

Review Comment:
   Confirmed and fixed in ab7c85da8f6. `JsonProcessingException` now logs the source and physical line number before the same exception is rethrown. Tests cover both malformed JSON and a mapping error on physical line 4 after two skipped lines.
   
   Truncated-last-line handling and ordinary stream IOException propagation are preserved. A separate test checks that a stream read failure does not receive a stale JSON parse-line diagnostic. Removing the new JSON diagnostic makes its regression test fail.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,8 +167,8 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
         "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
         "this to 0 or a negative value disables the limit. " +

Review Comment:
   Corrected the configuration docs and helper comment in ab7c85da8f6. They now state that non-positive values and values above `Int.MaxValue` use the maximum supported limit of 2,147,483,647 bytes. The existing configuration normalization is unchanged. Added tests for zero, negative, exact-maximum and larger values.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val bus = new ReplayListenerBus(8 * 1024)
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles UTF-8 boundaries and line endings") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    // scalastyle:off nonascii
+    val names = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")

Review Comment:
   Confirmed: Jackson escapes U+1F600 in the original test. Replaced that grid in ab7c85da8f6 with raw UTF-8 input observed through the replay filter before JSON parsing.
   
   The final code point is 1–4 bytes wide and starts at offsets 8, 8190, 8191 or 8192. Limits range from three bytes below to two bytes above the content length, covering positions inside multibyte code points and sequences split across the reader's byte buffer. LF, CRLF and unterminated final lines are included. A mutant counting supplementary characters as 3+3 bytes fails the new grid.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4118186744


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(maxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
-            }
+            buffer.append(c.toChar)
             c = reader.read()
           }
-          if (overLong) {
+          val line = buffer.result()
+          if (line.isEmpty) {

Review Comment:
   nit: `line` is an `Option[String]` here, so `line.isEmpty` reads like an empty-string check at first glance (and an empty line is in fact accepted as `Some("")`). Something like `buffer.result() match { case Some(line) => (line, index); case None => ... }`, or naming it `maybeLine`, would make the skip branch clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4136184468


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   Actually, I'm not sure about that. Both different versions and `3.5.10` sounds misleading to me. However, if there is any precedence, I don't care.
   
   > Note that I found the backport PRs of [SPARK-59407](https://issues.apache.org/jira/browse/SPARK-59407) fills this .version field of this config with the version number in each branch, i.e., .version("4.1.4") on branch-4.1 backport, .version("3.5.10") on branch-3.5 backport.
   
   cc @holdenk for looping the author of that backport.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4117452237


##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val bus = new ReplayListenerBus(8 * 1024)
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles UTF-8 boundaries and line endings") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    // scalastyle:off nonascii
+    val names = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      name <- names
+      repetitions <- Seq(100, 4097)
+      ending <- Seq("\n", "\r\n", "")
+      excess <- Seq(0, 1)
+    } {
+      val start = SparkListenerApplicationStart(name * repetitions, None, 125L, "user", None)
+      val json = JsonProtocol.sparkEventToJsonString(start)
+      val limit = json.getBytes(StandardCharsets.UTF_8).length - excess
+      val listener = new EventBufferingListener
+      val bus = new ReplayListenerBus(limit)
+      bus.addListener(listener)
+      // Put a short event first, including when the final line has no terminator.
+      val input = new ByteArrayInputStream((end + "\n" + json + ending)
+        .getBytes(StandardCharsets.UTF_8))
+      assert(bus.replay(input, "utf8"))
+      val expected = if (excess == 0) Seq(end, json) else Seq(end)
+      assert(listener.loggedEvents.toSeq == expected, s"name=$name ending=$ending excess=$excess")

Review Comment:
   Updated in ab7c85da8f6. Both replay success and observed-line assertions are wrapped in a clue containing the code point, padding, numeric line-ending bytes and limit delta. The old repetitions grid is replaced by deliberate boundary offsets.
   
   I retained the short ASCII application-event test because it now also exercises the `8k` configuration and successful JSON event delivery. The raw UTF-8 grid deliberately bypasses JSON parsing so serialization cannot hide the byte sequences being tested.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {

Review Comment:
   Confirmed by rerunning all five tests from the initial PR revision against the original implementation: four fail, and the malformed-UTF-8 test passes. The earlier count of three came from a run before the truncated/filtered test was added. I removed the stale count from the PR description.
   
   The revised suite has 15 passing tests, and both production and test scalastyle checks pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4118214576


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(maxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
-            }
+            buffer.append(c.toChar)
             c = reader.read()
           }
-          if (overLong) {
+          val line = buffer.result()
+          if (line.isEmpty) {

Review Comment:
   Renamed it to `maybeLine` in aaee04effe9 to make the `Option[String]` semantics clear. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4119567878


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -242,4 +248,37 @@ private[spark] object ReplayListenerBus {
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()
+    private var byteLength = 0L
+    private val bufferLimit = maxLineLength.toLong + 1

Review Comment:
   I reproduced the 600m / 620,000,000 ASCII characters + U+0100 case with the actual reader on JDK 21 and -Xmx3g. The effective content limit is now capped at 512m in both configuration resolution and the constructor, and the same input is successfully skipped. Tests cover normalization and the docs distinguish this JVM limit from heap requirements.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,18 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 2147483647, use the maximum supported limit of " +
+        "2147483647 bytes. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .bytesConf(ByteUnit.BYTE)
-      .createWithDefaultString("512m")
+      .createWithDefaultString("256m")

Review Comment:
   Confirmed: v4.3.0-rc1 does not contain SPARK-59407, while branch-4.3 does. This fix needs to reach branch-4.3 before the next RC, along with the other affected branches. Branch-3.5 needs a separate backport adapting the logging API.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -202,6 +205,10 @@ private[spark] class ReplayListenerBus(
         // Just stop replay.
         false
       case _: EOFException if maybeTruncated => false
+      case jpe: JsonProcessingException =>
+        logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)} " +
+          log"at line ${MDC(LINE_NUM, lineNumber)}", jpe)

Review Comment:
   Removed the throwable from the location diagnostic and kept the same exception rethrow. The tests now assert a single location diagnostic with getThrown == null for both parsing and mapping errors.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,149 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == Int.MaxValue)
+    }
+  }
+
+  test("Replay preserves physical line numbers after skipping long lines") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      (end + "\n" + "x" * 2048 + "\n" + "x" * 2048 + "\n{}\n")
+        .getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(!bus.replay(input, "line-numbers"))
+    }
+    assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Malformed line #4: {}")))
+  }
+
+  test("Replay logs physical line numbers before rethrowing JSON errors") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val mappingError =
+      """{"Event":"org.apache.spark.util.TestListenerEvent","foo":"x","bar":[]}"""
+    for ((malformed, errorClass) <- Seq(
+        ("{bad", classOf[JsonParseException]),
+        (mappingError, classOf[MismatchedInputException]))) {
+      val appender = new LogAppender
+      val bus = new ReplayListenerBus(1024)
+      val input = new ByteArrayInputStream(
+        Seq(end, "x" * 2048, "x" * 2048, malformed, end).mkString("\n")
+          .getBytes(StandardCharsets.UTF_8))
+      withLogAppender(appender) {
+        val error = intercept[JsonProcessingException] {
+          bus.replay(input, "json-errors", maybeTruncated = true)
+        }
+        assert(errorClass.isInstance(error))
+      }
+      assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+        .contains("Exception parsing Spark event log: json-errors at line 4")))
+    }
+  }
+
+  test("Replay propagates stream IO errors without reporting a stale parse line") {
+    val error = new IOException("read failure")
+    val input = new InputStream {
+      override def read(): Int = throw error
+    }
+    val appender = new LogAppender
+    withLogAppender(appender) {
+      assert(intercept[IOException] {
+        new ReplayListenerBus().replay(input, "read-error")
+      } eq error)
+    }
+    assert(!appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Exception parsing Spark event log")))
+  }
+
+  test("Replay preserves line numbers for truncated logs and filtered iterators") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val truncated = "{\"Event\":"
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      ("x" * 2048 + "\n" + end + "\n" + truncated).getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(bus.replay(input, "truncated", maybeTruncated = true,
+        eventsFilter = _ != end))
+      assert(bus.replay(Iterator(end, end, truncated), "iterator", maybeTruncated = true,
+        eventsFilter = _ != end))
+    }
+    val warnings = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+      .filter(_.contains("Got JsonParseException"))
+    assert(warnings.size == 2)
+    assert(warnings.forall(_.contains("at line 3,")))
+  }
+
+  test("Replay still rejects malformed UTF-8 in skipped lines") {
+    val bytes = Array.fill[Byte](20 * 1024)('x'.toByte) ++ Array(0xff.toByte, '\n'.toByte)
+    val bus = new ReplayListenerBus(1024)
+    intercept[java.nio.charset.MalformedInputException] {
+      bus.replay(new ByteArrayInputStream(bytes), "invalid-utf8")
+    }
+  }
+
   /**
    * Test replaying compressed spark history file that internally throws an EOFException.  To

Review Comment:
   Confirmed the pre-existing assertion gaps. I have narrowed the testing description; we will fix the replay result, event count/content assertions and log-file selection in a separate test follow-up.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +145,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)

Review Comment:
   Confirmed the skip-relative numbering and reproduced source.next() failing when the offset lands inside a UTF-8 sequence. We will cover both in the tail-reparse follow-up, including discarding the partial line as raw bytes before decoding and correcting the diagnostic location. The SHS listing/retry behavior still needs an integration regression test.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +66,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(
+      logData: InputStream,
+      sourceName: String): Iterator[(String, Int)] = {
     // Fail on malformed input like Source.getLines() does instead of replacing it.
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)

Review Comment:
   Reproduced: an ASCII-truncated final JSON line is tolerated with maybeTruncated, while a partial UTF-8 code point at EOF throws MalformedInputException. We will handle this separately, including compressed-log regression coverage.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -226,13 +233,12 @@ private[spark] class HaltReplayException extends RuntimeException
 private[spark] object ReplayListenerBus {
 
   /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
+   * Default UTF-8 line-length cap, matching spark.history.fs.eventLog.maxLineLength.
+   * Keep the maximum buffered character count close to the original 512 MiB / 2 cap.
    */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  val DEFAULT_MAX_LINE_LENGTH: Int = 256 * 1024 * 1024

Review Comment:
   The reader default now comes from History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue. I also replaced the historical comment with a description of the shared default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5908815330

   Regarding version, I think the convention is the lowest version that ships the config, identical on every branch including master. So 3.5.10, on all branches


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5894424040

   > Shall we mark the ticket as a blocker (rather than minor) if we intend to block the release? Also does this warrant blocking Apache Spark 4.3.0 as well? If it does, please reply to the mail thread named "Blockers for Apache Spark 4.3.0 RC".
   
   Makes sense. I set this as a blocker. I will reply the mail thread.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5889896277

   Shall we mark the ticket as a blocker (rather than minor) if we intend to block the release? Also does this warrant blocking Apache Spark 4.3.0 as well? If it does, please reply to the mail thread named "Blockers for Apache Spark 4.3.0 RC".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5916421742

   Thanks @dongjoon-hyun ! Do you have any more comments on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #59074:
URL: https://github.com/apache/spark/pull/59074#issuecomment-5916047847

   > Regarding version, I think the convention is the lowest version that ships the config, identical on every branch including master. So 3.5.10, on all branches
   
   I don't have strong opinion on this. Actually I prefer not to change it because it is not what this bug fix proposes to fix. And, there seems no consensus on this.
   
   I prefer to focus on the bug in this PR and leave the config version unchanged, so we can unblock the RCs quickly. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4117057071


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true

Review Comment:
   Test coverage: because L148 now re-decides from the full `byteLength`, this cap is no longer observable by any test. A mutant that always appends (never sets `overLong`), or one that uses `2 * maxLineLength + 1`, produces the same lines, indices, warnings and exceptions for every input in the suite; only the buffered size of skipped lines changes (e.g. 8193 -> 9216 chars in "Over-long event log lines are skipped instead of materialized"). Before this PR, removing the cap made that test fail. A test that observes the buffer bound would guard SPARK-59407's main guarantee, e.g. by extracting the reader into a helper and asserting the max buffered length, or by streaming a huge line from a lazy `InputStream` with a small limit.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {

Review Comment:
   This changes the per-line memory bound, not only the unit. Previously the builder stopped growing at `maxLineLength / 2` chars; now it can hold `maxLineLength + 1` chars, and a single char >= U+0100 inflates it to UTF-16 (2 bytes per char). With the unchanged default of `512m`:
   
   - Default SHS heap (`-Xmx1g`), one line of U+0101 followed by 600,000,000 ASCII chars: the current code stops at capacity 301,989,886 chars and skips the line. With this PR the builder grows to capacity 603,979,774 chars (a ~1.2 GB array) and fails with `OutOfMemoryError: Java heap space` before the line is rejected. `case e: Exception` does not catch it.
   - For configured limits in [576m, 1152m), a Latin-1 builder that has grown past 603,979,774 chars has capacity 1,207,959,550. `AbstractStringBuilder.inflate()` sizes the UTF-16 array from the capacity, so the first char >= U+0100 fails with `UTF16 String size is 1207959550, should be less than 1073741823` regardless of the heap size.
   - On JDK 8 (branch-3.5) there are no compact strings, so even pure-ASCII lines need about twice the heap of the current code.
   
   The docs still say the setting bounds "the memory replay can use". Could we default to `256m` (keeps the current bound for ASCII lines), keep a separate heap-oriented cap for the buffer, or document that heap usage can reach about 2-4x the limit?



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true
+              }
             }
             c = reader.read()

Review Comment:
   Performance (pre-existing since SPARK-59407, kept by this rewrite): `BufferedReader.read()` takes a lock on every call (on JDK 21 an `InternalLock`/`ReentrantLock`, whose CAS survives C2 inlining). On ~100 MB of TaskEnd-like ASCII lines this loop splits at ~73 MB/s (13.6 s/GB) vs ~1,039 MB/s (0.96 s/GB) for the previous `Source.getLines()`; together with Jackson `readTree` it is 60.9 MB/s vs 293 MB/s. That is roughly +12.6 s of CPU per GB for UI rebuilds and compaction, and ~8 s to drain a 600 MiB line. The UTF-8 width branching added here is not measurable. Keeping the same per-char logic but reading into a local `char[]` via `InputStreamReader.read(char[], ...)` is ~5x faster, and bulk-scanning the buffer for `\n` with slice appends is ~15x faster. Maybe a follow-up?



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {
+                1
+              } else if (c < 0x800) {
+                2
+              } else if (Character.isHighSurrogate(c.toChar)) {
+                4
+              } else if (Character.isLowSurrogate(c.toChar)) {
+                0
+              } else {
+                3
+              })
+              // Allow one extra byte until we know whether the line ends in CRLF.
+              if (byteLength <= maxLineLength.toLong + 1) {
+                sb.append(c.toChar)
+              } else {
+                overLong = true
+              }
             }
             c = reader.read()
           }
-          if (overLong) {
+          if (sb.length() > 0 && sb.charAt(sb.length() - 1) == '\r') {
+            sb.setLength(sb.length() - 1)
+            byteLength -= 1
+          }
+          if (overLong || byteLength > maxLineLength) {

Review Comment:
   nit: `overLong ||` is redundant. `overLong` is only set when `byteLength >= maxLineLength + 2`, `byteLength` no longer changes afterwards, and the CR strip subtracts at most 1, so `byteLength > maxLineLength` already holds (also for `Int.MaxValue`, 0 and negative limits; checked exhaustively for small limits plus random inputs). Either use `if (byteLength > maxLineLength)`, or drop the flag and keep only `if (byteLength <= maxLineLength + 1L) sb.append(c.toChar)` in the loop, which removes one nesting level.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +170,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)

Review Comment:
   This overload is also what the end-event reparse in `FsHistoryProvider` uses (FsHistoryProvider.scala L1109-L1127), fed by an unbounded `Source.fromInputStream(in)(Codec.UTF8).getLines()`. So on that path there is no `maxLineLength` and line numbers are relative to the skip point. With the defaults (compressed event logs, `endEventReparseChunkSize=1m`), completed apps always take this path, and because `target = lastFile.getLen - reparseChunkSize` is based on the compressed length while `skip` runs on the decompressed stream, most of the log is read there. In a local repro (156.6 MB decompressed / 16.4 MB compressed), 141 MB went through `getLines()`, and a 12.6 MB line over a 4 MiB limit was fully materialized. This predates the PR, but since the PR defines the limit and line-number semantics, it may be worth reusing the bounded reader there (skip, drop the partial first line, then replay through `boundedLines`) or filing a follow-up.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +107,54 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
+      @tailrec private def fetchLine(): (String, Int) = {
         val sb = new java.lang.StringBuilder()
         var overLong = false
+        var byteLength = 0L
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
+            if (!overLong) {
+              // The strict decoder guarantees valid surrogate pairs. Count their four UTF-8
+              // bytes on the high surrogate and zero on the low surrogate.
+              byteLength += (if (c < 0x80) {

Review Comment:
   nit: since the strict decoder only emits complete surrogate pairs, counting 2 bytes per surrogate half gives identical results (the running total matches at every code point boundary), so this could be `if (c < 0x80) 1 else if (c < 0x800 || Character.isSurrogate(c.toChar)) 2 else 3`, and the comment above becomes simpler.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +66,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(

Review Comment:
   Since `boundedLines` is private, compaction reads the same files with two different readers: `EventLogFileCompactor.initializeBuilders` replays through this bounded reader, while `EventFilter.applyFilterToFile` (EventFilter.scala L79) rewrites them with an unbounded `Source.getLines()`. If a skipped over-limit line is the start event of a still-live entity (e.g. `StageSubmitted` of a running stage, `JobStart` of a running job, `SQLExecutionStart` of a running execution without jobs), the builders never mark it live, the rewrite materializes the line and rejects it together with its dependent task events, and `cleanupCompactedFiles` deletes the originals, so raising the limit later cannot recover them. A port of the builder/filter logic lost 23 events with a 1 MiB limit and 2 MiB lines. The inconsistency came with SPARK-59407 and this PR only moves the threshold (e.g. CJK-heavy lines between limit/3 and limit/2 chars are newly skipped), but sharing one bounded reader for both passe
 s, and passing over-long lines through unchanged in the rewrite, would make them consistent.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,8 +167,8 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
         "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
         "this to 0 or a negative value disables the limit. " +

Review Comment:
   With the new `Long` arithmetic, "disables the limit" is no longer strictly true. `ReplayListenerBus.maxLineLength` maps `<= 0` (and values above `Int.MaxValue`) to `Int.MaxValue`, and lines with more than 2,147,483,647 UTF-8 bytes are now skipped with a warning (e.g. 2^30 x U+00E9 is 2,147,483,648 bytes but only a 1 GiB Latin-1 builder), whereas the previous `sb.length() * 2 < Int.MaxValue` overflowed and never skipped. Spark 3.4+ writers cannot produce such a line (one `ByteArrayOutputStream` per event), so this is mostly theoretical, but skipping the check when the limit is disabled, or documenting the 2^31-1 byte cap, would keep the doc accurate.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {

Review Comment:
   minor, about the PR description: it says "Three new regression tests fail against the original implementation", but four of the five new tests fail on the current code (this one, the boundary grid, the physical-line-number test and the truncated/filtered test); only "Replay still rejects malformed UTF-8 in skipped lines" passes there. Since the description becomes the commit message, it may be worth correcting.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -148,15 +170,21 @@ private[spark] class ReplayListenerBus(
       sourceName: String,
       maybeTruncated: Boolean,
       eventsFilter: ReplayEventsFilter): Boolean = {
+    replayEntries(lines.zipWithIndex, sourceName, maybeTruncated, eventsFilter)
+  }
+
+  private def replayEntries(

Review Comment:
   The physical line number only surfaces for non-IOException failures. `JsonParseException` (and `JsonMappingException`s such as `MismatchedInputException`) extend `IOException`, so for a malformed JSON line in the middle of a file, `throw jpe` (L219) is caught by `case ioe: IOException => throw ioe` (L233) before the `Malformed line #N` log (L237). For example, `[end, 2048 x's, 2048 x's, {bad, end]` with a 1024 limit throws without any line number, and `FsHistoryProvider` only logs Jackson's `line: 1, column: N`, which is relative to the single-line string. `{}` in the new test goes through the NPE path. The case ordering is old (SPARK-2261), but logging `sourceName`/`lineNumber` before rethrowing would make this fix effective for the most common corruption.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val bus = new ReplayListenerBus(8 * 1024)
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles UTF-8 boundaries and line endings") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    // scalastyle:off nonascii
+    val names = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")

Review Comment:
   The U+1F600 case does not exercise the 4-byte path. `JsonProtocol` serializes through Jackson's `UTF8JsonGenerator`, which writes supplementary characters as the ASCII escape `\ud83d\ude00` (`COMBINE_UNICODE_SURROGATES_IN_UTF8` is off by default in 2.22.2). So that JSON line is pure ASCII, the surrogate branches (ReplayListenerBus.scala L128-L131) are never taken, and mutants that count 3+3, 3+0 or 0+0 bytes for a pair all pass. Also, the boundary always falls on the trailing ASCII `","Timestamp":125,"User":"user"}`, so no multi-byte character straddles the limit, and since the name starts at byte offset 110, the 8192-byte decoder boundary never splits a U+00E9/U+4E2D sequence (8192 - 110 = 8082 is divisible by 2 and 3). Could we build raw UTF-8 input directly (e.g. lines whose last code point is 2, 3 or 4 bytes) and put the limit at -1/0/+1/+2 bytes around it?



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +109,85 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val bus = new ReplayListenerBus(8 * 1024)
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles UTF-8 boundaries and line endings") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    // scalastyle:off nonascii
+    val names = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      name <- names
+      repetitions <- Seq(100, 4097)
+      ending <- Seq("\n", "\r\n", "")
+      excess <- Seq(0, 1)
+    } {
+      val start = SparkListenerApplicationStart(name * repetitions, None, 125L, "user", None)
+      val json = JsonProtocol.sparkEventToJsonString(start)
+      val limit = json.getBytes(StandardCharsets.UTF_8).length - excess
+      val listener = new EventBufferingListener
+      val bus = new ReplayListenerBus(limit)
+      bus.addListener(listener)
+      // Put a short event first, including when the final line has no terminator.
+      val input = new ByteArrayInputStream((end + "\n" + json + ending)
+        .getBytes(StandardCharsets.UTF_8))
+      assert(bus.replay(input, "utf8"))
+      val expected = if (excess == 0) Seq(end, json) else Seq(end)
+      assert(listener.loggedEvents.toSeq == expected, s"name=$name ending=$ending excess=$excess")

Review Comment:
   nit: (1) The clue omits `repetitions` and prints `ending` raw, so `\n` and `\r\n` failures look identical, e.g. `s"name=$name repetitions=$repetitions ending=${ending.map(_.toInt).mkString(",")} excess=$excess"`; the `assert(bus.replay(input, "utf8"))` above has no clue at all. (2) `repetitions` 100 vs 4097 gives the same outcome for every (name, ending, excess) combination, so it only doubles the grid (see the decoder-boundary note on L127). (3) "Replay line limit uses UTF-8 bytes" is subsumed by the `name = "x"`, `excess = 0` iterations here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4125737022


##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -35,15 +35,17 @@ import org.apache.spark.util.JsonProtocol
 /**
  * A SparkListenerBus that can be used to replay events from serialized event data.
  *
- * @param maxLineLength Maximum number of byes (1/2 chars) of a single event log line that will be
- *                      materialized during replay. Longer lines are drained, skipped and
- *                      logged, bounding the memory replay can use when an event log is
- *                      corrupt or unexpectedly large.
+ * @param maxLineLength Maximum UTF-8 byte length of a single event log line, excluding its line
+ *                      ending. Longer lines are drained, skipped and logged, bounding the
+ *                      memory replay can use when an event log is corrupt or unexpectedly large.
  */
 private[spark] class ReplayListenerBus(

Review Comment:
   Binary compatibility (pre-existing from SPARK-59407, but this PR is a good place to fix it before the maintenance releases): every released version (3.5.9, 4.0.4, 4.1.3, 4.2.0) declares `class ReplayListenerBus extends SparkListenerBus with Logging`, which has a public no-arg constructor. With `(maxLineLength: Int = DEFAULT_MAX_LINE_LENGTH)`, scalac only emits `<init>(I)V`; default arguments don't generate a no-arg constructor.
   
   The class is `private[spark]` and excluded from MiMa, but event-log tools already use it and would break on 3.5.10 / 4.0.5 / 4.1.4 / 4.2.1:
   - Sparklens: `Class.forName("org.apache.spark.scheduler.ReplayListenerBus")` + `newInstance()` ([EventHistoryReporter.scala#L20-L21](https://github.com/qubole/sparklens/blob/7f2ea4bac91eff4435923881c263e6d42caa0949/src/main/scala/com/qubole/sparklens/app/EventHistoryReporter.scala#L20-L21)) -> `InstantiationException`.
   - aws-emr-advisor: `getDeclaredConstructor().newInstance()` ([EmrSparkLogParser.scala#L75-L76](https://github.com/aws-samples/aws-emr-advisor/blob/3caea29121322c7e76b61443694355f80d66bf6c/src/main/scala/com/amazonaws/emr/spark/EmrSparkLogParser.scala#L75-L76)) -> `NoSuchMethodException`, and `class CustomReplayListenerBus extends ReplayListenerBus` ([CustomReplayListenerBus.scala#L25](https://github.com/aws-samples/aws-emr-advisor/blob/3caea29121322c7e76b61443694355f80d66bf6c/src/main/scala/org/apache/spark/utils/CustomReplayListenerBus.scala#L25)) -> `NoSuchMethodError: 'void <init>()'` when compiled against a released version.
   
   The reflective cases fail even after recompiling. Could we add an auxiliary constructor?
   ```scala
     def this() = this(ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
   ```
   It compiles on Scala 2.12 and 2.13 with or without the default argument, `new ReplayListenerBus()` is not ambiguous, and all in-repo call shapes still compile, including `new ReplayListenerBus(maxLineLength = ...)` and the anonymous subclass in `StreamingQueryListenerSuite`. If 4.2.1 ships from RC1, keeping the default argument as well also preserves `$lessinit$greater$default$1` for code compiled against 4.2.1.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "512m", "536870913", "576m", "600m", "1g",
+        "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.MAX_LINE_LENGTH)
+    }
+  }
+
+  test("Replay constructor normalizes the effective line limit") {

Review Comment:
   Test coverage: this only reads `effectiveMaxLineLength`, and no test replays through a bus built with an out-of-range limit. So reverting `ReplayListenerBus` L114 to the raw parameter (`new BoundedLineBuffer(maxLineLength)`, as in the base code) passes the whole suite, although `new ReplayListenerBus(0)` would then skip every non-empty line, and `-1` / `Int.MinValue` every line. All production callers pass the already-normalized `maxLineLength(conf)`, so there's no user impact today, but the hand-ported branch-3.5 backport could regress without a failing test. Could we replay one event through the `0`, `-1` and `Int.MinValue` buses in the first loop and assert that it arrives?



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -225,21 +235,56 @@ private[spark] class HaltReplayException extends RuntimeException
 
 private[spark] object ReplayListenerBus {
 
-  /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
-   */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  /** Default UTF-8 content limit shared with the history server configuration. */
+  val DEFAULT_MAX_LINE_LENGTH: Int = History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue.get.toInt
+
+  // Bound StringBuilder growth so UTF-16 inflation stays below the JVM array-size limit.
+  val MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
 
-  /** Resolves the replay line-length cap from configuration; <= 0 disables the cap. */
+  /** Resolves the byte limit, using 512 MiB for non-positive or larger values. */
   def maxLineLength(conf: SparkConf): Int = {
-    val configured = conf.get(History.EVENT_LOG_MAX_LINE_LENGTH)
-    if (configured <= 0 || configured > Int.MaxValue) Int.MaxValue else configured.toInt
+    normalizeMaxLineLength(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH))
+  }
+
+  private def normalizeMaxLineLength(configured: Long): Int = {
+    if (configured <= 0 || configured > MAX_LINE_LENGTH) MAX_LINE_LENGTH else configured.toInt
   }
 
   type ReplayEventsFilter = (String) => Boolean
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()
+    private var byteLength = 0L
+    private val bufferLimit = maxLineLength.toLong + 1
+
+    def length: Int = buffer.length()
+
+    def append(c: Char): Unit = {
+      if (byteLength <= bufferLimit) {
+        // The strict decoder emits valid surrogate pairs: count two bytes per half.
+        byteLength += (if (c < 0x80) 1 else if (c < 0x800 || Character.isSurrogate(c)) 2 else 3)
+        // Reserve one extra byte for a possible CR in the line ending.
+        if (byteLength <= bufferLimit) {

Review Comment:
   Once `byteLength` exceeds `bufferLimit`, the line is certain to be skipped, but the retained prefix stays reachable until the `\n` is read. With the default `256m`, that is ~302 MB (Latin-1) or ~604 MB (UTF-16, i.e. any char >= U+0100 early in the line) of dead heap for the whole drain (~13 s per GB at the current read rate), overlapping with concurrent UI loads and compaction. In a `-Xmx1g` model, bus A draining U+0101 + ~868 MB of ASCII made bus B fail with `OutOfMemoryError` on a 200 MB line, and releasing the buffer on overflow made it pass.
   
   This is pre-existing from SPARK-59407, but it's a small fix:
   ```scala
           if (byteLength <= bufferLimit) {
             buffer.append(c)
           } else {
             // The line will be skipped; release the retained prefix while draining.
             buffer.setLength(0)
             buffer.trimToSize()
           }
   ```
   After overflow `byteLength >= maxLineLength + 2`, so `result()` still returns `None` regardless of a trailing CR. It doesn't lower the peak, only how long the memory is held. The assertion at `ReplayListenerSuite` L164 (`buffer.length == limit + 1`) would become `== 0`.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -225,21 +235,56 @@ private[spark] class HaltReplayException extends RuntimeException
 
 private[spark] object ReplayListenerBus {
 
-  /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
-   */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  /** Default UTF-8 content limit shared with the history server configuration. */
+  val DEFAULT_MAX_LINE_LENGTH: Int = History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue.get.toInt
+
+  // Bound StringBuilder growth so UTF-16 inflation stays below the JVM array-size limit.
+  val MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
 
-  /** Resolves the replay line-length cap from configuration; <= 0 disables the cap. */
+  /** Resolves the byte limit, using 512 MiB for non-positive or larger values. */
   def maxLineLength(conf: SparkConf): Int = {
-    val configured = conf.get(History.EVENT_LOG_MAX_LINE_LENGTH)
-    if (configured <= 0 || configured > Int.MaxValue) Int.MaxValue else configured.toInt
+    normalizeMaxLineLength(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH))
+  }
+
+  private def normalizeMaxLineLength(configured: Long): Int = {
+    if (configured <= 0 || configured > MAX_LINE_LENGTH) MAX_LINE_LENGTH else configured.toInt
   }
 
   type ReplayEventsFilter = (String) => Boolean
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()

Review Comment:
   FYI, because this `StringBuilder` starts at 16 and grows as `2c + 2`, the retained buffer is a step function of the limit rather than proportional to it. For ASCII-heavy lines, every limit from 150,994,942 to 301,989,885 bytes (~144m to ~288m) ends with the same 301,989,886-char buffer as the `256m` default, and ~288m to 512m all end with 603,979,774 chars.
   
   So if an operator hits an OOM while a long line is drained and lowers the limit to `200m` or `150m`, nothing changes. With `-Xmx800m`, U+0101 followed by 300,000,000 ASCII chars failed at the same point (length 150,994,942, when the 302 MB UTF-16 array is copied into a 604 MB one) for `256m`, `200m` and `150m`, while `143m` and `128m` skipped it. Could the docs mention that only limits below ~144m (e.g. `128m`) reduce the buffer, or could the growth be capped at `bufferLimit`?



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -202,6 +208,10 @@ private[spark] class ReplayListenerBus(
         // Just stop replay.
         false
       case _: EOFException if maybeTruncated => false
+      case jpe: JsonProcessingException =>
+        logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)} " +
+          log"at line ${MDC(LINE_NUM, lineNumber)}")
+        throw jpe
       case ioe: IOException =>
         throw ioe
       case e: Exception =>

Review Comment:
   Pre-existing and optional: the new JSON branch above logs only the source and line, but this generic branch still logs the entire line (`Malformed line #...: ${MDC(LINE, currentLine)}` at L219), which can be up to 256 MiB by default (512 MiB max). A large line that is valid JSON but semantically wrong ends up here, e.g. a `JobStart` whose `Properties` contain a numeric value fails `extractString`'s `require` with `IllegalArgumentException`. With the same code in Spark 3.5.0, a 33,554,544-char line produced a single 33,554,606-char ERROR record, and building the message temporarily needs about twice the line size (the `log"..."` builder doubles when the trailing `\n` is appended). Since this PR is about bounding replay memory, could we log a bounded prefix plus the line length instead? The `Malformed line #4: {}` assertion would still pass.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -67,22 +69,26 @@ private[spark] class ReplayListenerBus(
       maybeTruncated: Boolean = false,
       eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Boolean = {
     val lines = boundedLines(logData, sourceName)
-    replay(lines, sourceName, maybeTruncated, eventsFilter)
+    replayEntries(lines, sourceName, maybeTruncated, eventsFilter)
   }
 
   /**
-   * Reads '\n'-terminated lines like Source.getLines(), but never materializes more than
-   * `maxLineLength` bytes of a single line. An over-long line is drained and skipped
-   * with a warning instead of being turned into a String.
+   * Reads '\n'-terminated lines and retains their original zero-based indices. The limit
+   * measures UTF-8 content bytes, excluding the line ending, rather than JVM heap usage.
+   * The character buffer is bounded by the limit plus one possible trailing CR. An over-long
+   * line is drained and skipped with a warning instead of being turned into a String.
    */
-  private def boundedLines(logData: InputStream, sourceName: String): Iterator[String] = {
+  private def boundedLines(
+      logData: InputStream,
+      sourceName: String): Iterator[(String, Int)] = {
     // Fail on malformed input like Source.getLines() does instead of replacing it.
     val decoder = StandardCharsets.UTF_8.newDecoder()
       .onMalformedInput(CodingErrorAction.REPORT)
       .onUnmappableCharacter(CodingErrorAction.REPORT)
     val reader = new BufferedReader(new InputStreamReader(logData, decoder))

Review Comment:
   A thought for the block-read follow-up, not a blocker here: the limit is a UTF-8 byte count, but the reader decodes to UTF-16 first and re-derives the byte width per char. Since `0x0A` never appears inside a multi-byte UTF-8 sequence, splitting lines on raw bytes into a buffer capped at `limit + 1` bytes, and strictly decoding only the lines that are kept, makes the unit exact by construction. Skipped bytes can still be streamed through the strict decoder to keep the "malformed UTF-8 in skipped lines" contract. It would also:
   - bound a skipped non-Latin-1 line to `limit + 1` bytes instead of a UTF-16 builder (~604 MB plus growth at `256m`),
   - let a `MalformedInputException` be attributed to a physical line (today the decoder's 8 KiB read-ahead can throw before the preceding lines are delivered),
   - provide the raw-byte handling that the tail-reparse (partial first line) and truncated-UTF-8-at-EOF follow-ups need.
   
   A quick Java port matched this reader on all the grids in this PR plus random and malformed inputs. The reader alone ran at ~2,450 / ~310 MB/s (ASCII / CJK), vs ~77 / ~91 MB/s here and ~1,140 / ~215 MB/s for a bulk `char[]` reader. It still needs the CR reservation, a cap (around the 1 GiB UTF-16 `String` limit), and a heap-oriented default.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)

Review Comment:
   For the block-read follow-up: a new `BoundedLineBuffer` (and a 16-char `StringBuilder` regrown by doubling) per line, including skipped lines, allocates ~4.3 bytes per input byte on the event logs under `core/src/test/resources/spark-events`, vs ~1.7 with one buffer per iterator that is reset for each line (Jackson `readTree` alone is ~3.9). Today the per-char `BufferedReader.read()` lock hides the CPU cost, but with a bulk reader it becomes ~10% of the read loop. Most of this is inherited from SPARK-59407 (this PR adds ~40 B per line), so it's fine to handle it there. A reused buffer would need a capacity cap so that it doesn't pin a huge array for the rest of the file.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,19 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 536870912, use the maximum supported limit of " +

Review Comment:
   nit: This hard-codes `536870912` twice, duplicating `ReplayListenerBus.MAX_LINE_LENGTH`, similar to my earlier comment about `DEFAULT_MAX_LINE_LENGTH`. We could define the cap in `History` above `EVENT_LOG_MAX_LINE_LENGTH` (like `DEFAULT_LOG_DIR`), interpolate it with `.doc(s"...")`, and derive `ReplayListenerBus.MAX_LINE_LENGTH` from it. The other direction (referencing `ReplayListenerBus.MAX_LINE_LENGTH` from this doc) would create an initialization cycle, because the `ReplayListenerBus` object already reads `History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue` during its initialization. `monitoring.md` would still need a manual update.



##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,19 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 536870912, use the maximum supported limit of " +
+        "536870912 bytes (512 MiB). This cap avoids JVM array-size limits but does not " +
+        "guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +

Review Comment:
   Following up on my earlier backport comment: `v4.2.1-rc1` (cut 2026-09-28 06:11 UTC, about an hour before the latest commit here) already contains SPARK-59407 with the old semantics: `.version("4.2.1")`, the `512m` default, the `sb.length() * 2` check, and "Setting this to 0 or a negative value disables the limit". `branch-4.2` has already moved to `4.2.2-SNAPSHOT` without this fix. My earlier comment only covered `branch-4.3`, assuming nothing had shipped yet.
   
   If 4.2.1 is finalized from RC1 and this PR is backported afterwards, a patch release (4.2.2) would change the unit, the default and the `<= 0` meaning of a released config:
   - Default users: the same for ASCII, but stricter for non-ASCII-heavy lines (e.g. 268M -> 134M chars for 2-byte text, 268M -> ~89.5M chars for 3-byte text).
   - `8k`: an `ApplicationStart` line with 3,000 x U+4E2D (3,085 chars / 9,085 bytes) is replayed by RC1 but skipped here, so the app is missing from the listing.
   - `0` / negative / `3g`: RC1 never skips (`sb.length() * 2 < Int.MaxValue` is always true), while this PR clamps to 512 MiB, so there is no way to replay events above 512 MiB anymore.
   - An explicit `512m` (RC1's documented default): RC1 keeps at most 268,435,456 chars and skips U+0101 followed by 600,000,000 ASCII chars with `-Xmx1g`, while this PR grows the builder to 603,979,774 chars (~1.2 GB as UTF-16) and fails with `OutOfMemoryError` on UI load or compaction.
   
   Could we get this into `branch-4.2` before 4.2.1 is finalized, e.g. by raising it on the RC1 vote so that it lands in RC2? Otherwise, we need a `core-migration-guide.md` entry for 4.2.1 -> 4.2.2 and should qualify "also available in ... 4.2.1" here and in `monitoring.md`.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "512m", "536870913", "576m", "600m", "1g",
+        "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.MAX_LINE_LENGTH)
+    }
+  }
+
+  test("Replay constructor normalizes the effective line limit") {
+    val maximum = ReplayListenerBus.MAX_LINE_LENGTH
+    for (limit <- Seq(Int.MinValue, -1, 0, maximum, maximum + 1, Int.MaxValue)) {
+      assert(new ReplayListenerBus(limit).effectiveMaxLineLength == maximum)
+    }
+    for (limit <- Seq(1, 8192, ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH, maximum - 1)) {
+      assert(new ReplayListenerBus(limit).effectiveMaxLineLength == limit)
+      val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH, limit.toLong)
+      assert(ReplayListenerBus.maxLineLength(conf) == limit)
+    }
+    assert(new ReplayListenerBus().effectiveMaxLineLength ==
+      ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+  }
+
+  test("Replay preserves physical line numbers after skipping long lines") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      (end + "\n" + "x" * 2048 + "\n" + "x" * 2048 + "\n{}\n")
+        .getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(!bus.replay(input, "line-numbers"))
+    }
+    assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Malformed line #4: {}")))
+  }
+
+  test("Replay logs physical line numbers before rethrowing JSON errors") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val mappingError =
+      """{"Event":"org.apache.spark.util.TestListenerEvent","foo":"x","bar":[]}"""
+    for ((malformed, errorClass) <- Seq(
+        ("{bad", classOf[JsonParseException]),
+        (mappingError, classOf[MismatchedInputException]))) {
+      val appender = new LogAppender
+      val bus = new ReplayListenerBus(1024)
+      val input = new ByteArrayInputStream(
+        Seq(end, "x" * 2048, "x" * 2048, malformed, end).mkString("\n")
+          .getBytes(StandardCharsets.UTF_8))
+      withLogAppender(appender) {
+        val error = intercept[JsonProcessingException] {
+          bus.replay(input, "json-errors", maybeTruncated = true)
+        }
+        assert(errorClass.isInstance(error))
+      }
+      val diagnostics = appender.loggingEvents.filter(_.getMessage.getFormattedMessage
+        .contains("Exception parsing Spark event log: json-errors at line 4"))
+      assert(diagnostics.size == 1)
+      assert(diagnostics.head.getThrown == null)
+    }
+  }
+
+  test("Replay propagates stream IO errors without reporting a stale parse line") {

Review Comment:
   This stream throws on the very first `read()`, so no line has been parsed and `currentLine` / `lineNumber` are still at their initial values; the stale-line situation in the test name never occurs. For example, a later change like `case ioe: IOException => if (currentLine != null) logError(log"... at line ${MDC(LINE_NUM, lineNumber)}")` would still pass, while it would report `at line 1` for a real log whose first event parses and whose compressed stream then fails. Could the stream serve one valid event line (e.g. `end + "\n"`) before throwing, keeping the `eq error` and no-diagnostic assertions?



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex

Review Comment:
   nit: This hand-rolls what `zipWithIndex` already does at L151. If the reader emits one `Option` per physical line, both overloads number lines the same way, and the manual counter, the tuple-typed state and the `@tailrec` recursion go away:
   ```scala
       var warned = false
       Iterator.continually(reader.read()).takeWhile(_ != -1).map { first =>
         val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
         var c = first
         while (c != -1 && c != '\n') {
           buffer.append(c.toChar)
           c = reader.read()
         }
         val maybeLine = buffer.result()
         if (maybeLine.isEmpty && !warned) {
           logWarning(...) // unchanged
           warned = true
         }
         maybeLine
       }.zipWithIndex.collect { case (Some(line), index) => (line, index) }
   ```
   A `NextIterator[Option[String]]` also works if relying on `takeWhile`'s one-element look-ahead feels too implicit. I checked that this behaves the same as the current iterator on Scala 2.12 and 2.13, including the look-ahead from `lineEntries.hasNext` in the `JsonParseException` handler and exceptions thrown mid-line. Feel free to skip it if you'd rather keep the diff against SPARK-59407 small.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
-            }
+            buffer.append(c.toChar)
             c = reader.read()
           }
-          if (overLong) {
+          val maybeLine = buffer.result()
+          if (maybeLine.isEmpty) {
             if (!warned) {
               logWarning(log"Skipped event log lines longer than " +

Review Comment:
   Since the physical `index` is available here now, could the warning include it (e.g. `index + 1` of the first skipped line)? Today a replay can return `true` after dropping several lines, and the only trace is one "Skipped event log lines longer than 268435456 bytes in <path>" per file, without a location or a count; later skips are not logged at any level. `replayEntries` already uses one WARN plus a per-occurrence `logDebug` for unrecognized events (L176-L183), which would fit here too. A count at EOF alone wouldn't be enough, because listing replays often stop early with `HaltReplayException`.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {

Review Comment:
   nit: Since this PR fixes a specific issue and adds tests to an existing suite, could we prefix the new test names with the JIRA ID, e.g. `test("SPARK-59804: Replay line limit uses UTF-8 bytes")`, as the contributing guide suggests? The same applies to the other new tests (L128, L157, L169, L180, L194, L208, L233, L248, L267). It helps to trace failures across the backport branches.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -35,15 +35,17 @@ import org.apache.spark.util.JsonProtocol
 /**
  * A SparkListenerBus that can be used to replay events from serialized event data.
  *
- * @param maxLineLength Maximum number of byes (1/2 chars) of a single event log line that will be
- *                      materialized during replay. Longer lines are drained, skipped and
- *                      logged, bounding the memory replay can use when an event log is
- *                      corrupt or unexpectedly large.
+ * @param maxLineLength Maximum UTF-8 byte length of a single event log line, excluding its line

Review Comment:
   nit: Could the `@param` mention the normalization, e.g. "Values <= 0 or above `ReplayListenerBus.MAX_LINE_LENGTH` (512 MiB) use `MAX_LINE_LENGTH`"? Otherwise a caller passing `0` (RC1's "disables the limit" semantics) or `1 << 30` silently gets a 512 MiB limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4126667357


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -167,16 +167,19 @@ private[spark] object History {
 
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Values at or below 0, or above 536870912, use the maximum supported limit of " +
+        "536870912 bytes (512 MiB). This cap avoids JVM array-size limits but does not " +
+        "guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +

Review Comment:
   Confirmed that v4.2.1-rc1 contains the old implementation and branch-4.2 is now 4.2.2-SNAPSHOT. I raised this on the Spark dev mailing list so we can decide whether to produce an RC2 with the reviewed fix before finalizing 4.2.1. I am holding off on migration changes pending that discussion.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -35,15 +35,17 @@ import org.apache.spark.util.JsonProtocol
 /**
  * A SparkListenerBus that can be used to replay events from serialized event data.
  *
- * @param maxLineLength Maximum number of byes (1/2 chars) of a single event log line that will be
- *                      materialized during replay. Longer lines are drained, skipped and
- *                      logged, bounding the memory replay can use when an event log is
- *                      corrupt or unexpectedly large.
+ * @param maxLineLength Maximum UTF-8 byte length of a single event log line, excluding its line
+ *                      ending. Longer lines are drained, skipped and logged, bounding the
+ *                      memory replay can use when an event log is corrupt or unexpectedly large.
  */
 private[spark] class ReplayListenerBus(

Review Comment:
   Confirmed with the compiled class: the no-argument constructor was missing. I restored it while retaining the default argument, and added a reflection regression test. The compiled class now exposes both constructors and the Scala default-argument method.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -225,21 +235,56 @@ private[spark] class HaltReplayException extends RuntimeException
 
 private[spark] object ReplayListenerBus {
 
-  /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
-   */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  /** Default UTF-8 content limit shared with the history server configuration. */
+  val DEFAULT_MAX_LINE_LENGTH: Int = History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue.get.toInt
+
+  // Bound StringBuilder growth so UTF-16 inflation stays below the JVM array-size limit.
+  val MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
 
-  /** Resolves the replay line-length cap from configuration; <= 0 disables the cap. */
+  /** Resolves the byte limit, using 512 MiB for non-positive or larger values. */
   def maxLineLength(conf: SparkConf): Int = {
-    val configured = conf.get(History.EVENT_LOG_MAX_LINE_LENGTH)
-    if (configured <= 0 || configured > Int.MaxValue) Int.MaxValue else configured.toInt
+    normalizeMaxLineLength(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH))
+  }
+
+  private def normalizeMaxLineLength(configured: Long): Int = {
+    if (configured <= 0 || configured > MAX_LINE_LENGTH) MAX_LINE_LENGTH else configured.toInt
   }
 
   type ReplayEventsFilter = (String) => Boolean
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()
+    private var byteLength = 0L
+    private val bufferLimit = maxLineLength.toLong + 1
+
+    def length: Int = buffer.length()
+
+    def append(c: Char): Unit = {
+      if (byteLength <= bufferLimit) {
+        // The strict decoder emits valid surrogate pairs: count two bytes per half.
+        byteLength += (if (c < 0x80) 1 else if (c < 0x800 || Character.isSurrogate(c)) 2 else 3)
+        // Reserve one extra byte for a possible CR in the line ending.
+        if (byteLength <= bufferLimit) {

Review Comment:
   The retained prefix is now cleared and trimmed once overflow makes the line unconditionally skippable. The buffer test checks both zero length and zero capacity after overflow; this reduces retention during draining, not the allocation peak.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -225,21 +235,56 @@ private[spark] class HaltReplayException extends RuntimeException
 
 private[spark] object ReplayListenerBus {
 
-  /**
-   * Default per-line cap during replay: far above any legitimate event line, so replay
-   * memory stays bounded even for corrupt logs. Matches the default of
-   * spark.history.fs.eventLog.maxLineLength.
-   */
-  val DEFAULT_MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
+  /** Default UTF-8 content limit shared with the history server configuration. */
+  val DEFAULT_MAX_LINE_LENGTH: Int = History.EVENT_LOG_MAX_LINE_LENGTH.defaultValue.get.toInt
+
+  // Bound StringBuilder growth so UTF-16 inflation stays below the JVM array-size limit.
+  val MAX_LINE_LENGTH: Int = 512 * 1024 * 1024
 
-  /** Resolves the replay line-length cap from configuration; <= 0 disables the cap. */
+  /** Resolves the byte limit, using 512 MiB for non-positive or larger values. */
   def maxLineLength(conf: SparkConf): Int = {
-    val configured = conf.get(History.EVENT_LOG_MAX_LINE_LENGTH)
-    if (configured <= 0 || configured > Int.MaxValue) Int.MaxValue else configured.toInt
+    normalizeMaxLineLength(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH))
+  }
+
+  private def normalizeMaxLineLength(configured: Long): Int = {
+    if (configured <= 0 || configured > MAX_LINE_LENGTH) MAX_LINE_LENGTH else configured.toInt
   }
 
   type ReplayEventsFilter = (String) => Boolean
 
   // utility filter that selects all event logs during replay
   val SELECT_ALL_FILTER: ReplayEventsFilter = { (eventString: String) => true }
+
+  /** Accumulates a bounded prefix of one decoded line, allowing a possible trailing CR. */
+  private[scheduler] class BoundedLineBuffer(maxLineLength: Int) {
+    private val buffer = new java.lang.StringBuilder()

Review Comment:
   Confirmed the capacity steps from the JDK growth rule. I documented that lowering 256m to 200m or 150m may leave the same buffer capacity, with 128m as an example of a smaller step. Changing buffer growth remains part of the reader follow-up.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "512m", "536870913", "576m", "600m", "1g",
+        "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.MAX_LINE_LENGTH)
+    }
+  }
+
+  test("Replay constructor normalizes the effective line limit") {

Review Comment:
   The constructor test now replays an event and checks delivery for every out-of-range argument in the loop, including 0, -1 and Int.MinValue.



##########
core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala:
##########
@@ -109,6 +112,166 @@ class ReplayListenerSuite extends SparkFunSuite with BeforeAndAfter with LocalSp
     assert(eventMonster.loggedEvents(1) === JsonProtocol.sparkEventToJsonString(applicationEnd))
   }
 
+  test("Replay line limit uses UTF-8 bytes") {
+    val start = SparkListenerApplicationStart("x" * 6000, None, 125L, "user", None)
+    val json = JsonProtocol.sparkEventToJsonString(start)
+    assert(json.getBytes(StandardCharsets.UTF_8).length < 8 * 1024)
+    val listener = new EventBufferingListener
+    val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH.key, "8k")
+    val bus = new ReplayListenerBus(ReplayListenerBus.maxLineLength(conf))
+    bus.addListener(listener)
+    val input = new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))
+    assert(bus.replay(input, "ascii"))
+    assert(listener.loggedEvents.toSeq == Seq(json))
+  }
+
+  test("Replay line limit handles raw UTF-8 boundaries and line endings") {
+    // scalastyle:off nonascii
+    val suffixes = Seq("x", "\u00e9", "\u4e2d", "\ud83d\ude00")
+    // scalastyle:on nonascii
+    for {
+      suffix <- suffixes
+      padding <- Seq(8, 8190, 8191, 8192)
+      ending <- Seq("\n", "\r\n", "")
+      delta <- -3 to 2
+    } {
+      // Put the limit inside the final code point, and split UTF-8 sequences across read buffers.
+      val line = "x" * padding + suffix
+      val bytes = line.getBytes(StandardCharsets.UTF_8)
+      val limit = bytes.length + delta
+      val seen = new ArrayBuffer[String]
+      val bus = new ReplayListenerBus(limit)
+      val input = new ByteArrayInputStream((line + ending).getBytes(StandardCharsets.UTF_8))
+      withClue(s"codePoint=${suffix.codePointAt(0)} padding=$padding " +
+          s"ending=${ending.map(_.toInt).mkString(",")} delta=$delta: ") {
+        // Observe raw lines before JSON parsing, without letting Jackson escape the test input.
+        assert(bus.replay(input, "utf8", eventsFilter = line => {
+          seen += line
+          false
+        }))
+        assert(seen.toSeq == (if (delta >= 0) Seq(line) else Seq.empty))
+      }
+    }
+  }
+
+  test("Replay line buffer stops retaining oversized content") {
+    for (limit <- Seq(0, 1, 8, 1024)) {
+      val buffer = new ReplayListenerBus.BoundedLineBuffer(limit)
+      for (_ <- 0 until 10000) {
+        buffer.append('x')
+        assert(buffer.length <= limit + 1)
+      }
+      assert(buffer.length == limit + 1)
+      assert(buffer.result().isEmpty)
+    }
+  }
+
+  test("Replay line limit defaults and maximum supported configuration") {
+    val conf = new SparkConf(false)
+    assert(conf.get(History.EVENT_LOG_MAX_LINE_LENGTH) == 256L * 1024 * 1024)
+    assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+    for (value <- Seq("0", "-1", "512m", "536870913", "576m", "600m", "1g",
+        "2147483647", "3g")) {
+      conf.set(History.EVENT_LOG_MAX_LINE_LENGTH.key, value)
+      assert(ReplayListenerBus.maxLineLength(conf) == ReplayListenerBus.MAX_LINE_LENGTH)
+    }
+  }
+
+  test("Replay constructor normalizes the effective line limit") {
+    val maximum = ReplayListenerBus.MAX_LINE_LENGTH
+    for (limit <- Seq(Int.MinValue, -1, 0, maximum, maximum + 1, Int.MaxValue)) {
+      assert(new ReplayListenerBus(limit).effectiveMaxLineLength == maximum)
+    }
+    for (limit <- Seq(1, 8192, ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH, maximum - 1)) {
+      assert(new ReplayListenerBus(limit).effectiveMaxLineLength == limit)
+      val conf = new SparkConf(false).set(History.EVENT_LOG_MAX_LINE_LENGTH, limit.toLong)
+      assert(ReplayListenerBus.maxLineLength(conf) == limit)
+    }
+    assert(new ReplayListenerBus().effectiveMaxLineLength ==
+      ReplayListenerBus.DEFAULT_MAX_LINE_LENGTH)
+  }
+
+  test("Replay preserves physical line numbers after skipping long lines") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val appender = new LogAppender
+    val bus = new ReplayListenerBus(1024)
+    val input = new ByteArrayInputStream(
+      (end + "\n" + "x" * 2048 + "\n" + "x" * 2048 + "\n{}\n")
+        .getBytes(StandardCharsets.UTF_8))
+    withLogAppender(appender) {
+      assert(!bus.replay(input, "line-numbers"))
+    }
+    assert(appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+      .contains("Malformed line #4: {}")))
+  }
+
+  test("Replay logs physical line numbers before rethrowing JSON errors") {
+    val end = JsonProtocol.sparkEventToJsonString(SparkListenerApplicationEnd(1000L))
+    val mappingError =
+      """{"Event":"org.apache.spark.util.TestListenerEvent","foo":"x","bar":[]}"""
+    for ((malformed, errorClass) <- Seq(
+        ("{bad", classOf[JsonParseException]),
+        (mappingError, classOf[MismatchedInputException]))) {
+      val appender = new LogAppender
+      val bus = new ReplayListenerBus(1024)
+      val input = new ByteArrayInputStream(
+        Seq(end, "x" * 2048, "x" * 2048, malformed, end).mkString("\n")
+          .getBytes(StandardCharsets.UTF_8))
+      withLogAppender(appender) {
+        val error = intercept[JsonProcessingException] {
+          bus.replay(input, "json-errors", maybeTruncated = true)
+        }
+        assert(errorClass.isInstance(error))
+      }
+      val diagnostics = appender.loggingEvents.filter(_.getMessage.getFormattedMessage
+        .contains("Exception parsing Spark event log: json-errors at line 4"))
+      assert(diagnostics.size == 1)
+      assert(diagnostics.head.getThrown == null)
+    }
+  }
+
+  test("Replay propagates stream IO errors without reporting a stale parse line") {

Review Comment:
   The stream now serves one complete event before throwing. The test verifies that the event was delivered, the original IOException is rethrown, and no stale parsing diagnostic is emitted.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -104,35 +110,29 @@ private[spark] class ReplayListenerBus(
         line
       }
 
-      @tailrec private def fetchLine(): String = {
-        val sb = new java.lang.StringBuilder()
-        var overLong = false
+      @tailrec private def fetchLine(): (String, Int) = {
+        val buffer = new BoundedLineBuffer(effectiveMaxLineLength)
         var c = reader.read()
         if (c == -1) {
           null
         } else {
+          val index = lineIndex
+          lineIndex += 1
           while (c != -1 && c != '\n') {
-            if (sb.length() * 2 < maxLineLength) {
-              sb.append(c.toChar)
-            } else {
-              overLong = true
-            }
+            buffer.append(c.toChar)
             c = reader.read()
           }
-          if (overLong) {
+          val maybeLine = buffer.result()
+          if (maybeLine.isEmpty) {
             if (!warned) {
               logWarning(log"Skipped event log lines longer than " +

Review Comment:
   Added the first skipped physical line number to the warning and a DEBUG message for every skipped line. A regression test checks one warning and the locations of both skipped lines.



##########
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:
##########
@@ -202,6 +208,10 @@ private[spark] class ReplayListenerBus(
         // Just stop replay.
         false
       case _: EOFException if maybeTruncated => false
+      case jpe: JsonProcessingException =>
+        logError(log"Exception parsing Spark event log: ${MDC(PATH, sourceName)} " +
+          log"at line ${MDC(LINE_NUM, lineNumber)}")
+        throw jpe
       case ioe: IOException =>
         throw ioe
       case e: Exception =>

Review Comment:
   The malformed-line diagnostic now includes at most the first 1,024 characters and reports the full character count. A regression test uses valid JSON with a numeric property value to exercise the generic exception path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4131900489


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   I think we have done this with 3.5.10 in this case so far IIRC



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59804][UI] Fix UTF-8 byte limits and physical line numbers during event log replay [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4135994178


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
       .bytesConf(ByteUnit.BYTE)
       .createWithDefaultString("1m")
 
+  val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
   val EVENT_LOG_MAX_LINE_LENGTH =
     ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
-      .doc("Maximum length of a single event log line during replay. Lines longer than " +
-        "this are skipped with a warning instead of being read into memory, bounding the " +
-        "memory replay can use when an event log is corrupt or unexpectedly large. Setting " +
-        "this to 0 or a negative value disables the limit. " +
+      .doc("Maximum UTF-8 byte length of a single event log line during replay, excluding " +
+        "the line ending. Longer lines are skipped with a warning, bounding the " +
+        "memory replay can use when an event log is corrupt or unexpectedly large. Buffer " +
+        "growth, UTF-16 storage and JSON parsing can require several times this limit in heap " +
+        "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m may not reduce " +
+        "the retained buffer; use 128m to reach a smaller capacity. Values at or below 0, " +
+        s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported limit of " +
+        s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids JVM array-size " +
+        "limits but does not guarantee sufficient heap space. " +
         "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and 4.2.1; and in " +
         "all versions after 4.3.0.")
       .version("4.3.0")

Review Comment:
   Do you mean to set `.version("3.5.10")` for all branches?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org