You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nathan Kronenfeld <nk...@uncharted.software> on 2017/09/01 15:31:17 UTC
isCached
I'm currently porting some of our code from RDDs to Datasets.
With RDDs it's pretty easy to figure out if they are cached or not.
I notice that the catalog has a function for determining this on Datasets
too, but it's private[sql]. Is there any reason for it not to be public?
Is there any way at the moment to determine if a dataset is cached or not?
Thanks in advance
-Nathan Kronenfeld
Re: isCached
Posted by Nathan Kronenfeld <nk...@uncharted.software>.
Thanks for the info
On Fri, Sep 1, 2017 at 12:06 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> No unfortunately not - as i recall storageLevel accesses some private
> methods to get the result.
>
> On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld <nk...@uncharted.software>
> wrote:
>
>> Ah, in 2.1.0.
>>
>> I'm in 2.0.1 at the moment... is there any way that works that far back?
>>
>> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <nick.pentreath@gmail.com
>> > wrote:
>>
>>> Dataset does have storageLevel. So you can use isCached = (storageLevel
>>> != StorageLevel.NONE) as a test.
>>>
>>> Arguably isCached could be added to dataset too, shouldn't be a
>>> controversial change.
>>>
>>> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld <nk...@uncharted.software>
>>> wrote:
>>>
>>>> I'm currently porting some of our code from RDDs to Datasets.
>>>>
>>>> With RDDs it's pretty easy to figure out if they are cached or not.
>>>>
>>>> I notice that the catalog has a function for determining this on
>>>> Datasets too, but it's private[sql]. Is there any reason for it not to be
>>>> public? Is there any way at the moment to determine if a dataset is cached
>>>> or not?
>>>>
>>>> Thanks in advance
>>>> -Nathan Kronenfeld
>>>>
>>>
>>
Re: isCached
Posted by Nick Pentreath <ni...@gmail.com>.
No unfortunately not - as i recall storageLevel accesses some private
methods to get the result.
On Fri, 1 Sep 2017 at 17:55, Nathan Kronenfeld
<nk...@uncharted.software> wrote:
> Ah, in 2.1.0.
>
> I'm in 2.0.1 at the moment... is there any way that works that far back?
>
> On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <ni...@gmail.com>
> wrote:
>
>> Dataset does have storageLevel. So you can use isCached = (storageLevel
>> != StorageLevel.NONE) as a test.
>>
>> Arguably isCached could be added to dataset too, shouldn't be a
>> controversial change.
>>
>> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld
>> <nk...@uncharted.software> wrote:
>>
>>> I'm currently porting some of our code from RDDs to Datasets.
>>>
>>> With RDDs it's pretty easy to figure out if they are cached or not.
>>>
>>> I notice that the catalog has a function for determining this on
>>> Datasets too, but it's private[sql]. Is there any reason for it not to be
>>> public? Is there any way at the moment to determine if a dataset is cached
>>> or not?
>>>
>>> Thanks in advance
>>> -Nathan Kronenfeld
>>>
>>
>
Re: isCached
Posted by Nathan Kronenfeld <nk...@uncharted.software>.
Ah, in 2.1.0.
I'm in 2.0.1 at the moment... is there any way that works that far back?
On Fri, Sep 1, 2017 at 11:46 AM, Nick Pentreath <ni...@gmail.com>
wrote:
> Dataset does have storageLevel. So you can use isCached = (storageLevel !=
> StorageLevel.NONE) as a test.
>
> Arguably isCached could be added to dataset too, shouldn't be a
> controversial change.
>
> On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld <nk...@uncharted.software>
> wrote:
>
>> I'm currently porting some of our code from RDDs to Datasets.
>>
>> With RDDs it's pretty easy to figure out if they are cached or not.
>>
>> I notice that the catalog has a function for determining this on Datasets
>> too, but it's private[sql]. Is there any reason for it not to be public?
>> Is there any way at the moment to determine if a dataset is cached or not?
>>
>> Thanks in advance
>> -Nathan Kronenfeld
>>
>
Re: isCached
Posted by Nick Pentreath <ni...@gmail.com>.
Dataset does have storageLevel. So you can use isCached = (storageLevel !=
StorageLevel.NONE) as a test.
Arguably isCached could be added to dataset too, shouldn't be a
controversial change.
On Fri, 1 Sep 2017 at 17:31, Nathan Kronenfeld
<nk...@uncharted.software> wrote:
> I'm currently porting some of our code from RDDs to Datasets.
>
> With RDDs it's pretty easy to figure out if they are cached or not.
>
> I notice that the catalog has a function for determining this on Datasets
> too, but it's private[sql]. Is there any reason for it not to be public?
> Is there any way at the moment to determine if a dataset is cached or not?
>
> Thanks in advance
> -Nathan Kronenfeld
>