You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ajay Srivastava <a_...@yahoo.com> on 2014/07/04 08:19:20 UTC

OFF_HEAP storage level

Hi,
I was checking different storage level of an RDD and found OFF_HEAP.
Has anybody used this level ?
If i use this level, where will data be stored ? If not in heap, does it mean that we can avoid GC ?
How can I use this level ? I did not find anything in archive regarding this.
Can someone also explain the behavior of storage level - NONE ? 


Regards,
Ajay

Re: OFF_HEAP storage level

Posted by Ajay Srivastava <a_...@yahoo.com>.

Thanks Jerry.
It looks like a good option, will try it.

Regards,
Ajay



On Friday, July 4, 2014 2:18 PM, "Shao, Saisai" <sa...@intel.com> wrote:
 


Hi Ajay,
 
StorageLevel OFF_HEAP means for can cache your RDD into Tachyon, the prerequisite is that you should deploy Tachyon among Spark.
 
Yes, it can alleviate GC, since you offload JVM memory into system managed memory.
 
You can use rdd.persist(…) to use this level, details can be checked in BlockManager.scala, TachyonBlockManager and TachyonStore.
 
StorageLevel NONE means the rdd will not be cached, and if you want to use this rdd again, you should re-compute from the source to get the data.
 
Thanks
Jerry
 
From:Ajay Srivastava [mailto:a_k_srivastava@yahoo.com] 
Sent: Friday, July 04, 2014 2:19 PM
To: user@spark.apache.org
Subject: OFF_HEAP storage level
 
Hi,
I was checking different storage level of an RDD and found OFF_HEAP.
Has anybody used this level ?
If i use this level, where will data be stored ? If not in heap, does it mean that we can avoid GC ?
How can I use this level ? I did not find anything in archive regarding this.
Can someone also explain the behavior of storage level - NONE ? 


Regards,
Ajay

RE: OFF_HEAP storage level

Posted by "Shao, Saisai" <sa...@intel.com>.

Hi Ajay,

StorageLevel OFF_HEAP means for can cache your RDD into Tachyon, the prerequisite is that you should deploy Tachyon among Spark.

Yes, it can alleviate GC, since you offload JVM memory into system managed memory.

You can use rdd.persist(...) to use this level, details can be checked in BlockManager.scala, TachyonBlockManager and TachyonStore.

StorageLevel NONE means the rdd will not be cached, and if you want to use this rdd again, you should re-compute from the source to get the data.

Thanks
Jerry

From: Ajay Srivastava [mailto:a_k_srivastava@yahoo.com]
Sent: Friday, July 04, 2014 2:19 PM
To: user@spark.apache.org
Subject: OFF_HEAP storage level

Hi,
I was checking different storage level of an RDD and found OFF_HEAP.
Has anybody used this level ?
If i use this level, where will data be stored ? If not in heap, does it mean that we can avoid GC ?
How can I use this level ? I did not find anything in archive regarding this.
Can someone also explain the behavior of storage level - NONE ?


Regards,
Ajay