Spark Checkpointing behavior

[Tarek Elgamal <ta...@gmail.com>]

Hi,

I am trying to understand the behavior of rdd.checkpoint() in Spark. I am
running the JavaPageRank
<https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/JavaPageRank.java>
example on a 1 GB graph and I am checkpointing the *ranks *rdd inside each
iteration (between line 125 and 126 in the given link). Spark execution
starts when it hits the *collect()* action. I am expecting that after each
iteration the intermediate ranks will be materialized and written in the
checkpoint dir but, it seems that the rdd is only written once in the end
of the program, although I am invoking ranks.checkpoint() inside the for
loop. Is that the default behavior ?

Note that I am caching the rdd before checkpointing in order to avoid
recomputing

Best Regards,
Tarek