Spark Core

[arijit chakraborty <ak...@hotmail.com>]

Hi,


Suppose I've this following code:


a = matrix(seq(1,10), 10,1)


for(i in 1:100){

  b = a + 10

  write (b, "path" + ".csv", format="csv")

}


So what I'm doing is for 100 items, I'm adding a constant to a matrix than outputting it. And this operation occurs in spark using multiple core of the system.


My question is, after the operation is the value (here b) remains in that core (memory) of the system, so that it get piled up in the memory. Will this affect the performance of the process? If it is, how to clean the memory after each execution of loop?


The reason for asking the question is, when I'm testing the code in R the performance is much better than systemML. Since R to systemML is almost one-to-one mapping, so I'm not sure where I'm making the mistake. And unfortunately at the stage of progress I can't share the exact code.


Thanks you!

Arijit