How to release data frame to avoid memory leak

["kramer2009@126.com" <kr...@126.com>]

Hi

I have data frames created every 5 minutes. I use a dict to keep the recent
1 hour data frames. So only 12 data frame can be kept in the dict. New data
frame come in, old data frame pop out.

My question is when I pop out the old data frame, do I have to call
dataframe.unpersist to release the memory?

For example 

    If currentTime == fiveMinutes:

        myDict[currentTime] = dataframe

        oldestDataFrame = myDict.pop(oldest)

Now do I have to call oldestDataFrame.unpresist?  Because I think python
will automatically release unused variable






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-release-data-frame-to-avoid-memory-leak-tp26656.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org