You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "zhoukang (JIRA)" <ji...@apache.org> on 2017/05/04 01:45:04 UTC

[jira] [Comment Edited] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

    [ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996013#comment-15996013 ] 

zhoukang edited comment on SPARK-20582 at 5/4/17 1:44 AM:
----------------------------------------------------------

You are right [~vanzin],i have looked into you implementation.Thanks for your time.


was (Author: cane):
You are right [~vanzin],i searched into you implementation.Thanks for your time.

> Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-20582
>                 URL: https://issues.apache.org/jira/browse/SPARK-20582
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy
>    Affects Versions: 2.1.0
>            Reporter: zhoukang
>            Priority: Critical
>
>        In the current code of HistoryServer,jetty server will be started after all the logs fetch from yarn has been replayed.However,when the number of logs is becoming larger,the start time of jetty will be too long.
>        Here,we implement a solution which using ApplicationAttemptInfo checkpointing to speed up the start of historyserver.When historyserver is starting,it will load ApplicationAttemptInfo from checkpoint file first if exists which is faster then replaying one by one.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org