You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Judy Nash <ju...@exchange.microsoft.com> on 2014/12/13 00:14:21 UTC

Spark SQL API Doc & IsCached as SQL command

Hello,

Few questions on Spark SQL:


1)      Does Spark SQL support equivalent SQL Query for Scala command: IsCached(<table name>) ?


2)      Is there a documentation spec I can reference for question like this?



Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses


Thanks,
Judy

RE: Spark SQL API Doc & IsCached as SQL command

Posted by Judy Nash <ju...@exchange.microsoft.com>.

Thanks Cheng. Tried it out and saw the InMemoryColumnarTableScan word in the physical plan.

From: Cheng Lian [mailto:lian.cs.zju@gmail.com]
Sent: Friday, December 12, 2014 11:37 PM
To: Judy Nash; user@spark.apache.org
Subject: Re: Spark SQL API Doc & IsCached as SQL command

There isn’t a SQL statement that directly maps SQLContext.isCached, but you can use EXPLAIN EXTENDED to check whether the underlying physical plan is a InMemoryColumnarTableScan.

On 12/13/14 7:14 AM, Judy Nash wrote:
Hello,

Few questions on Spark SQL:

1)      Does Spark SQL support equivalent SQL Query for Scala command: IsCached(<table name>) ?

2)      Is there a documentation spec I can reference for question like this?

Closest doc I can find is this one: https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses

Thanks,
Judy

Re: Spark SQL API Doc & IsCached as SQL command

Posted by Cheng Lian <li...@gmail.com>.

There isn’t a SQL statement that directly maps |SQLContext.isCached|, 
but you can use |EXPLAIN EXTENDED| to check whether the underlying 
physical plan is a |InMemoryColumnarTableScan|.

On 12/13/14 7:14 AM, Judy Nash wrote:

> Hello,
>
> Few questions on Spark SQL:
>
> 1)Does Spark SQL support equivalent SQL Query for Scala command: 
> IsCached(<table name>) ?
>
> 2)Is there a documentation spec I can reference for question like this?
>
> Closest doc I can find is this one: 
> https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses
>
> Thanks,
>
> Judy
>

Re: Spark SQL API Doc & IsCached as SQL command

Posted by Mark Hamstra <ma...@clearstorydata.com>.

http://spark.apache.org/docs/latest/sql-programming-guide.html#caching-data-in-memory

On Fri, Dec 12, 2014 at 3:14 PM, Judy Nash <ju...@exchange.microsoft.com>
wrote:
>
>  Hello,
>
>
>
> Few questions on Spark SQL:
>
>
>
> 1)      Does Spark SQL support equivalent SQL Query for Scala command:
> IsCached(<table name>) ?
>
>
>
> 2)      Is there a documentation spec I can reference for question like
> this?
>
>
>
> Closest doc I can find is this one:
> https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#deploying-in-existing-hive-warehouses
>
>
>
> Thanks,
>
> Judy
>