You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Dong Wang (Jira)" <ji...@apache.org> on 2019/08/20 12:29:00 UTC

[jira] [Updated] (SPARK-28781) Unneccesary persist in PeriodicCheckpointer.update()

     [ https://issues.apache.org/jira/browse/SPARK-28781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dong Wang updated SPARK-28781:
------------------------------
    Environment:     (was: Once update(newData) is called, newData is persisted. However, only when the checkpoint is handling, the cached data is used for the second time (do the checkpoint job). The other data which is not satisfied to the checkpoint condition is actually unnecessary to be persisted. The persistedQueue avoids too much unnecessary cached data, but it is best to avoid every unnecessary persist operation.)

> Unneccesary persist in PeriodicCheckpointer.update()
> ----------------------------------------------------
>
>                 Key: SPARK-28781
>                 URL: https://issues.apache.org/jira/browse/SPARK-28781
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.3
>            Reporter: Dong Wang
>            Priority: Major
>
> * Once the update is called, newData is persisted. However, only when the checkpoint is handling, the persist data is used for the second time. The other data which is not satisfied to the checkpoint condition is unnecessary to be cached. The persistedQueue avoids too many unnecessary cached data, but it is best to avoid every unnecessary persist operation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org