[jira] [Created] (SPARK-28781) Unneccesary persist in PeriodicCheckpointer.update()

["Dong Wang (Jira)" <ji...@apache.org>]

Dong Wang created SPARK-28781:
---------------------------------

             Summary: Unneccesary persist in PeriodicCheckpointer.update()
                 Key: SPARK-28781
                 URL: https://issues.apache.org/jira/browse/SPARK-28781
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.3
         Environment: Once update(newData) is called, newData is persisted. However, only when the checkpoint is handling, the cached data is used for the second time (do the checkpoint job). The other data which is not satisfied to the checkpoint condition is actually unnecessary to be persisted. The persistedQueue avoids too much unnecessary cached data, but it is best to avoid every unnecessary persist operation.
            Reporter: Dong Wang


* Once the update is called, newData is persisted. However, only when the checkpoint is handling, the persist data is used for the second time. The other data which is not satisfied to the checkpoint condition is unnecessary to be cached. The persistedQueue avoids too many unnecessary cached data, but it is best to avoid every unnecessary persist operation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org