You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sung Hwan Chung <co...@cs.stanford.edu> on 2014/10/10 18:15:11 UTC
Spark job (not Spark streaming) doesn't delete un-needed checkpoints.
Un-needed checkpoints are not getting automatically deleted in my
application.
I.e. the lineage looks something like this and checkpoints simply
accumulate in a temporary directory (every lineage point, however, does zip
with a globally permanent):
PermanentRDD: Global zips with all the intermediate ones
Intermediate RDDs: A--->B--->C---->D---->E---->F---->----->G
| |
|
checkpoint checkpoint
checkpoint
Older intermediate RDDs never get used.