You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "holdenk (via GitHub)" <gi...@apache.org> on 2026/09/25 21:36:56 UTC
[PR] [SPARK-59407][UI][4.2] Improve compressed line handling in Spark History server [spark]
holdenk opened a new pull request, #59056:
URL: https://github.com/apache/spark/pull/59056
### What changes were proposed in this pull request?
Backport of SPARK-59407 to branch-4.2, based on master commit 38258cbea90c7540d71db86a64cfd18ccc0dffec (#58700).
Skip excessively long event log lines during History Server replay: `ReplayListenerBus` reads lines with a per-line byte cap (`spark.history.fs.eventLog.maxLineLength`, default 512m) instead of materializing unbounded lines; over-long lines are drained and skipped with a warning.
Branch adaptation vs master: the config declares `.version("4.2.1")` and keeps `.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)` (available on 4.2+).
### Why are the changes needed?
Individual corrupt or unexpectedly large event log lines can make loading a previous Spark job in the History Server slow or exhaust memory; bounding the per-line length keeps replay memory bounded.
### Does this PR introduce _any_ user-facing change?
Yes: new configuration `spark.history.fs.eventLog.maxLineLength` (default 512m). Event log lines longer than the cap are skipped with a warning instead of being read into memory during replay; setting it to 0 or a negative value disables the limit.
### How was this patch tested?
New test in `ReplayListenerSuite` covering that an over-long line is skipped while neighboring events still replay. Full core module test suite run locally on this branch; the 8 Ssl* suites fail on pristine branch-4.2 on this machine too (netty-tcnative 2.0.76 environment issue, unrelated to this change).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude and Cursor
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59407][UI][4.2] Improve compressed line handling in Spark History server [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk closed pull request #59056: [SPARK-59407][UI][4.2] Improve compressed line handling in Spark History server
URL: https://github.com/apache/spark/pull/59056
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59407][UI][4.2] Improve compressed line handling in Spark History server [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #59056:
URL: https://github.com/apache/spark/pull/59056#issuecomment-5841126281
Merged to 4.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org