You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/11/19 05:26:00 UTC
[jira] [Resolved] (SPARK-29579) Guarantee compatibility of snapshot
(live entities, KVstore entities)
[ https://issues.apache.org/jira/browse/SPARK-29579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-29579.
----------------------------------
Resolution: Invalid
SPARK-29779 invalidates the needs of this. Closing.
> Guarantee compatibility of snapshot (live entities, KVstore entities)
> ---------------------------------------------------------------------
>
> Key: SPARK-29579
> URL: https://issues.apache.org/jira/browse/SPARK-29579
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> This issue is a follow-up issue after SPARK-29111 and SPARK-29261, which both issues WILL NOT guarantee compatibility.
> To safely clean up old event log files after snapshot has been written for these files, we have to ensure the snapshot file can restore the state as same as we replay from these event log files. The issue is on migrating to the newer Spark version - if snapshot is not readable due to incompatibility, the app cannot be read entirely as we've already removed old event log files. If we can guarantee compatibility we can move on to the next item, cleaning up old event log files to save space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org