You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2015/12/01 10:54:27 UTC

New to Spark

 Hi,
I am new to Spark.
I am trying to use spark-sql with SPARK CREATED and HIVE CREATED tables.
I have successfully made Hive metastore to be used by Spark.
In spark-sql I can see the DDL for Hive tables. However, when I do select count(1) from HIVE_TABLE it always returns zero rows.
If I create a table in spark as create table SPARK_TABLE as select * from HIVE_TABLE, the table schema is created but no data.
I can then use Hive to do INSET SELECT from HIVE_TABLE into SPARK_TABLE. That works.
I can then use spark-sql to query the table.
My questions:
   
   - Is this correct that spark-sql only sees data in spark created tables but not any data in Hive tables?
   - How can I make Spark read data from existing Hive tables.


Thanks

Re: New to Spark

Posted by Ted Yu <yu...@gmail.com>.

Have you tried the following command ?
REFRESH TABLE <table>

Cheers



On Tue, Dec 1, 2015 at 1:54 AM, Ashok Kumar <as...@yahoo.com.invalid>
wrote:

> Hi,
>
> I am new to Spark.
>
> I am trying to use spark-sql with SPARK CREATED and HIVE CREATED tables.
>
> I have successfully made Hive metastore to be used by Spark.
>
> In spark-sql I can see the DDL for Hive tables. However, when I do select
> count(1) from HIVE_TABLE it always returns zero rows.
>
> If I create a table in spark as create table SPARK_TABLE as select * from
> HIVE_TABLE, the table schema is created but no data.
>
> I can then use Hive to do INSET SELECT from HIVE_TABLE into SPARK_TABLE.
> That works.
>
> I can then use spark-sql to query the table.
>
> My questions:
>
>
>    1. Is this correct that spark-sql only sees data in spark created
>    tables but not any data in Hive tables?
>    2. How can I make Spark read data from existing Hive tables.
>
>
>
> Thanks
>

Re: New to Spark

Posted by "fightfate@163.com" <fi...@163.com>.

Hi，there 
Which version spark in your use case ? You made hive metastore to be used by Spark, 
that mean you can run sql queries over the current hive tables , right ? Or you just use 
local hive metastore embeded in spark sql side ? I think you need to provide more info
for your spark sql and hive config, that would help to locate root cause for the problem.

Best,
Sun.



fightfate@163.com
 
From: Ashok Kumar
Date: 2015-12-01 18:54
To: user@spark.apache.org
Subject: New to Spark

Hi,

I am new to Spark.

I am trying to use spark-sql with SPARK CREATED and HIVE CREATED tables.

I have successfully made Hive metastore to be used by Spark.

In spark-sql I can see the DDL for Hive tables. However, when I do select count(1) from HIVE_TABLE it always returns zero rows.

If I create a table in spark as create table SPARK_TABLE as select * from HIVE_TABLE, the table schema is created but no data.

I can then use Hive to do INSET SELECT from HIVE_TABLE into SPARK_TABLE. That works.

I can then use spark-sql to query the table.

My questions:

Is this correct that spark-sql only sees data in spark created tables but not any data in Hive tables?
How can I make Spark read data from existing Hive tables.


Thanks

New to Spark

Posted by Ashok Kumar <as...@yahoo.com.INVALID>.

  Hi,
I am new to Spark.
I am trying to use spark-sql with SPARK CREATED and HIVE CREATED tables.
I have successfully made Hive metastore to be used by Spark.
In spark-sql I can see the DDL for Hive tables. However, when I do select count(1) from HIVE_TABLE it always returns zero rows.
If I create a table in spark as create table SPARK_TABLE as select * from HIVE_TABLE, the table schema is created but no data.
I can then use Hive to do INSET SELECT from HIVE_TABLE into SPARK_TABLE. That works.
I can then use spark-sql to query the table.
My questions:
   
   - Is this correct that spark-sql only sees data in spark created tables but not any data in Hive tables?
   - How can I make Spark read data from existing Hive tables.


Thanks