You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "lokesh.gidra" <lo...@gmail.com> on 2014/07/26 21:53:45 UTC

Lot of object serialization even with MEMORY_ONLY

Hello,

I am executing the SparkPageRank example. It uses the "cache()" API for
persistence of RDDs. And if I am not wrong, it in turn uses MEMORY_ONLY
storage level. However, in oprofile report it shows a lot of activity in
writeObject0 function.

There is not even a single "Spilling in-memory..." message in the
output/log. This is because I am using a huge heap size of 120GB.

Can someone please tell me why do I see so much serialization happening,
even though MEMORY_ONLY storage level is used? The spark version that I am
using is 1.0.1


Thanks,
Lokesh



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Lot-of-object-serialization-even-with-MEMORY-ONLY-tp10722.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.