You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Nathan Kronenfeld <nk...@oculusinfo.com> on 2014/07/15 16:31:23 UTC

persistence state of an RDD

Is there a way of determining programatically the cache state of an RDD?
Not its storage level, but how much of it is actually cached?

Thanks,
                    -Nathan





-- 
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone:  +1-416-203-3003 x 238
Email:  nkronenfeld@oculusinfo.com

Re: persistence state of an RDD

Posted by Nathan Kronenfeld <nk...@oculusinfo.com>.

Thanks


On Tue, Jul 15, 2014 at 10:39 AM, Praveen Seluka <ps...@qubole.com> wrote:

> Nathan, you are looking for SparkContext.getRDDStorageInfo which returns
> the information on how much is cached.
>
>
> On Tue, Jul 15, 2014 at 8:01 PM, Nathan Kronenfeld <
> nkronenfeld@oculusinfo.com> wrote:
>
>> Is there a way of determining programatically the cache state of an RDD?
>> Not its storage level, but how much of it is actually cached?
>>
>> Thanks,
>>                     -Nathan
>>
>>
>>
>>
>>
>> --
>> Nathan Kronenfeld
>> Senior Visualization Developer
>> Oculus Info Inc
>> 2 Berkeley Street, Suite 600,
>> Toronto, Ontario M5A 4J5
>> Phone:  +1-416-203-3003 x 238
>> Email:  nkronenfeld@oculusinfo.com
>>
>
>


-- 
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone:  +1-416-203-3003 x 238
Email:  nkronenfeld@oculusinfo.com

Re: persistence state of an RDD

Posted by Praveen Seluka <ps...@qubole.com>.

Nathan, you are looking for SparkContext.getRDDStorageInfo which returns
the information on how much is cached.


On Tue, Jul 15, 2014 at 8:01 PM, Nathan Kronenfeld <
nkronenfeld@oculusinfo.com> wrote:

> Is there a way of determining programatically the cache state of an RDD?
> Not its storage level, but how much of it is actually cached?
>
> Thanks,
>                     -Nathan
>
>
>
>
>
> --
> Nathan Kronenfeld
> Senior Visualization Developer
> Oculus Info Inc
> 2 Berkeley Street, Suite 600,
> Toronto, Ontario M5A 4J5
> Phone:  +1-416-203-3003 x 238
> Email:  nkronenfeld@oculusinfo.com
>