You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/08/25 23:58:00 UTC

[jira] [Created] (SPARK-28870) Snapshot old event log files to support compaction

Jungtaek Lim created SPARK-28870:
------------------------------------

             Summary: Snapshot old event log files to support compaction
                 Key: SPARK-28870
                 URL: https://issues.apache.org/jira/browse/SPARK-28870
             Project: Spark
          Issue Type: Sub-task
          Components: Spark Core
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


This issue tracks the effort on compacting old event log files into snapshot and achieve both goals, 1) reduce overall event log file size 2) speed up replaying event logs. It also deals with cleaning event log files as snapshot will provide the safe way to clean up old event log files without losing ability to replay whole event logs.

This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled event log files.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org