You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vishnu Viswanath <vi...@gmail.com> on 2016/02/29 00:09:32 UTC
Question about MEOMORY_AND_DISK persistence
Hi All,
I have a question regarding Persistence (MEMORY_AND_DISK)
Suppose I am trying to persist an RDD which has 2 partitions and only 1
partition can be fit in memory completely but some part of partition 2 can
also be fit, will spark keep the portion of partition 2 in memory and rest
in disk, or will the whole 2nd partition be kept in disk.
Regards,
Vishnu
Re: Question about MEOMORY_AND_DISK persistence
Posted by Vishnu Viswanath <vi...@gmail.com>.
Thank you Ashwin.
On Sun, Feb 28, 2016 at 7:19 PM, Ashwin Giridharan <as...@gmail.com>
wrote:
> Hi Vishnu,
>
> A partition will either be in memory or in disk.
>
> -Ashwin
> On Feb 28, 2016 15:09, "Vishnu Viswanath" <vi...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have a question regarding Persistence (MEMORY_AND_DISK)
>>
>> Suppose I am trying to persist an RDD which has 2 partitions and only 1
>> partition can be fit in memory completely but some part of partition 2 can
>> also be fit, will spark keep the portion of partition 2 in memory and rest
>> in disk, or will the whole 2nd partition be kept in disk.
>>
>> Regards,
>> Vishnu
>>
>
--
Thanks and Regards,
Vishnu Viswanath,
*www.vishnuviswanath.com <http://www.vishnuviswanath.com>*
Re: Question about MEOMORY_AND_DISK persistence
Posted by Ashwin Giridharan <as...@gmail.com>.
Hi Vishnu,
A partition will either be in memory or in disk.
-Ashwin
On Feb 28, 2016 15:09, "Vishnu Viswanath" <vi...@gmail.com>
wrote:
> Hi All,
>
> I have a question regarding Persistence (MEMORY_AND_DISK)
>
> Suppose I am trying to persist an RDD which has 2 partitions and only 1
> partition can be fit in memory completely but some part of partition 2 can
> also be fit, will spark keep the portion of partition 2 in memory and rest
> in disk, or will the whole 2nd partition be kept in disk.
>
> Regards,
> Vishnu
>