You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/08/27 06:09:59 UTC
[GitHub] [spark] HeartSaVioR edited a comment on issue #25577:
[WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log
file in HistoryServer
HeartSaVioR edited a comment on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer
URL: https://github.com/apache/spark/pull/25577#issuecomment-525153007
Hi @Ngone51 , I came across this PR as I've been working on similar things, [SPARK-28594](https://issues.apache.org/jira/browse/SPARK-28594).
While the goals of both issues seem orthogonal (as you want to speed up current event log whereas I'm proposing another version of event log to support rolling), the basic approach looks similar, snapshotting ("checkpoint" in here). I've experimented various approaches of snapshotting in POC (https://github.com/HeartSaVioR/spark/commit/d2f4c8970eb87218dfc9f426ec8e611248ef8f33) and came up with the snapshot approach which is cross-compatible with any KVStore implementations with only small modification of KVStore interface.
Could you please go through the [design doc](https://docs.google.com/document/d/12bdCC4nA58uveRxpeo8k7kGOI2NRTXmXyBOweSi4YcY/edit?usp=sharing), especially how to snapshot the KVStore instance and the format of the snapshot file? The design doc is in reviewing, but given the piece is common thing across both goals, I'll work on snapshot part first if you think it looks good.
It would be also great if you could provide design doc and the plan for this issue so that we can evaluate this as well.
Thanks in advance!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org