You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2026/10/12 21:36:00 UTC
[jira] [Resolved] (SPARK-60158) Avoid per-character synchronized reads in the bounded event log line reader
[ https://issues.apache.org/jira/browse/SPARK-60158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh resolved SPARK-60158.
---------------------------------
Fix Version/s: 4.2.2
4.1.5
4.0.6
4.3.0
3.5.10
Resolution: Fixed
Issue resolved by pull request 59360
[https://github.com/apache/spark/pull/59360]
> Avoid per-character synchronized reads in the bounded event log line reader
> ---------------------------------------------------------------------------
>
> Key: SPARK-60158
> URL: https://issues.apache.org/jira/browse/SPARK-60158
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.2, 4.1.5, 4.0.6, 4.3.0, 3.5.10
>
>
> SPARK-59407 replaced {{Source.getLines()}} in {{ReplayListenerBus.replay(InputStream, ...)}} with a bounded line reader so that over-long event log lines are skipped instead of materialized (SPARK-59804 later fixed its byte accounting). The reader calls {{BufferedReader.read()}} once per character, and {{BufferedReader.read()}} takes a lock on every call, so splitting lines is several times slower than the previous {{readLine()}}-based reading. In the review of SPARK-59804 (https://github.com/apache/spark/pull/59074), line splitting was measured at about 14x slower.
> This slows down every replay through the {{InputStream}} overload: loading application UIs in the History Server, event log compaction, and listing with fast in-progress parsing. SPARK-60109 also routes the History Server end-event reparse through this reader. For compressed logs, that reparse reads most of the decompressed file, because the skip target is computed from the compressed length, so it makes the listing of completed applications slower as well. In a local measurement of that reparse path (210 MB of uncompressed event lines, lz4 and zstd), it took about 0.17 s with {{Source.getLines()}} and 0.9-1.05 s with the bounded reader.
> Proposed fix: read decoded characters in bulk (e.g. into a {{char[]}} buffer) and scan for '\n' there, instead of calling {{read()}} per character. Keep the current semantics: the UTF-8 content byte limit with the trailing-CR allowance, physical line indices including skipped lines, strict decoding, and releasing the buffer of an over-long line while it is drained. A prototype of the bulk scan brought the reparse path above back to about 0.17 s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org