You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Imran Rashid (Jira)" <ji...@apache.org> on 2019/10/11 19:55:00 UTC
[jira] [Commented] (SPARK-28867) InMemoryStore checkpoint to speed
up replay log file in HistoryServer
[ https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949767#comment-16949767 ]
Imran Rashid commented on SPARK-28867:
--------------------------------------
This is closely related to SPARK-20656. Its not *quite* a duplicate, because that was about reparsing the logs for the same application within the same SHS instance -- so the SHS still had whatever state stored in memory. Here you're also talking about speeding up parsing of those files even when the SHS is restarted, which also requires some way to restore any state across SHS restarts.
> InMemoryStore checkpoint to speed up replay log file in HistoryServer
> ---------------------------------------------------------------------
>
> Key: SPARK-28867
> URL: https://issues.apache.org/jira/browse/SPARK-28867
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: wuyi
> Priority: Major
>
> HistoryServer now could be very slow to replay a large log file at the first time and it always re-replay an inprogress log file after it changes. we could periodically checkpoint InMemoryStore to speed up replay log file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org