[jira] [Updated] (SPARK-59804) Fix UTF-8 byte limits and physical line numbers during event log replay

["L. C. Hsieh (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

L. C. Hsieh updated SPARK-59804:
--------------------------------
    Fix Version/s: 3.5.10

> Fix UTF-8 byte limits and physical line numbers during event log replay
> -----------------------------------------------------------------------
>
>                 Key: SPARK-59804
>                 URL: https://issues.apache.org/jira/browse/SPARK-59804
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 4.3.0, 4.2.1, 4.0.5, 4.1.4, 3.5.10, 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 4.3.0, 4.2.1, 4.0.5, 4.1.4, 3.5.10
>
>
> SPARK-59407 (commit 38258cbea90) introduced bounded event-log line reading in ReplayListenerBus. There are two related correctness issues:
> 1. spark.history.fs.eventLog.maxLineLength is a byte-valued configuration, but boundedLines compares StringBuilder.length() * 2 with the configured limit. This measures UTF-16 storage rather than UTF-8 input length. With an 8 KiB limit, a valid ASCII SparkListenerApplicationStart event containing a 6,000-character application name (approximately 6 KiB of JSON) is incorrectly skipped. Multibyte UTF-8 input also has inconsistent boundaries. A trailing CR in CRLF is counted before being removed.
> 2. boundedLines removes oversized lines before replay applies zipWithIndex. Consequently, subsequent parse diagnostics report renumbered positions instead of physical file lines. For example, a valid event on line 1, oversized lines 2 and 3, and an invalid event {} on line 4 produce "Malformed line #2" instead of "Malformed line #4".
> The proposed fix counts UTF-8 content bytes, excludes line endings from the limit, and carries original line indices through skipped lines. It preserves strict rejection of malformed UTF-8, filtered iterator behavior, and truncated-log handling. Buffer growth remains bounded by the configured line limit; the limit specifies encoded content length rather than an exact JVM heap budget.
> Local regression coverage includes ASCII content, 1- through 4-byte UTF-8 characters, exact and over-limit boundaries, decoder-buffer crossings, LF/CRLF/unterminated final lines, physical line numbers after multiple skipped lines, filtered iterators, truncated logs, and malformed UTF-8 in oversized lines.
> Validation: the original implementation fails three new regression tests. With the fix, all 11 tests in ReplayListenerSuite pass under SBT, and both production and test scalastyle checks pass.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org