pyspark cached dataframe shows deserialized at StorageLevel

[Mitsutoshi Kiuchi <m....@nifty.com>]

Hi,

Spark document describes "Since the data is always serialized on the  
Python side, all the constants use the serialized formats.".

http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.StorageLevel

But when I cached dataframe and looked StorageLevel, it shows that  
cached dataframe is deserialized. It looks wrong behavior against  
document. but does anyone please comment about it ?

Here is code to reproduce.

from pyspark.sql.functions import col, length
source = [
     ("frighten",), ("watch",), ("dish",),
     ("reflect",), ("serious",), ("summer",),
     ("embrace",), ("transition",), ("Venus",)]
dfa = spark.createDataFrame(source, ["word"]).withColumn("num",  
length(col("word")))
dfa2 = dfa.cache()

dfa.storageLevel  >>> shows "StorageLevel(True, True, False, True, 1)"
dfa2.storageLevel >>> same as above


Thanks for reading!

Mitsutoshi Kiuchi

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org