[jira] [Created] (SPARK-60110) Skip event log compaction when replay skips lines over the line-length limit

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-60110:
-----------------------------------

             Summary: Skip event log compaction when replay skips lines over the line-length limit
                 Key: SPARK-60110
                 URL: https://issues.apache.org/jira/browse/SPARK-60110
             Project: Spark
          Issue Type: Bug
          Components: Web UI
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


SPARK-59407 bounded event log replay with spark.history.fs.eventLog.maxLineLength, but event log compaction only applies the limit in its first phase. EventLogFileCompactor.initializeBuilders replays through ReplayListenerBus, which skips over-long lines, while EventFilter.applyFilterToFile reads every line unbounded during rewrite.

When a skipped line is a live job's SparkListenerJobStart (or a live SparkListenerSQLExecutionStart), the filter builders never learn the job is live, so the rewrite drops the job start and all of its stage and task events, and the original files are then deleted, losing the data permanently. The rewrite phase also bypasses the memory bound.

This only happens when compaction is enabled (spark.history.fs.eventLog.rolling.maxFilesToRetain) and a line exceeds the limit.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org