You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Divya Gehlot <di...@gmail.com> on 2015/12/08 07:12:28 UTC
Unable to acces hive table (created through hive context) in hive console
Hi,
I am new bee to Spark and using HDP 2.2 which comes with Spark 1.3.1
I tried following code example
> import org.apache.spark.sql.SQLContext
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
>
> val personFile = "/user/hdfs/TestSpark/Person.csv"
> val df = sqlContext.load(
> "com.databricks.spark.csv",
> Map("path" -> personFile, "header" -> "true", "inferSchema" -> "true"))
> df.printSchema()
> val selectedData = df.select("Name", "Age")
> selectedData.save("NewPerson.csv", "com.databricks.spark.csv")
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
> hiveContext.sql("CREATE TABLE IF NOT EXISTS PersonTable (Name STRING, Age
> STRING)")
> hiveContext.sql("LOAD DATA INPATH '/user/hdfs/NewPerson.csv' INTO TABLE
> PersonTable")
> hiveContext.sql("from PersonTable SELECT Name, Age
> ").collect.foreach(println)
I am able to access above table in HDFS
> [hdfs@sandbox ~]$ hadoop fs -ls /user/hive/warehouse/persontable
> Found 3 items
> -rw-r--r-- 1 hdfs hdfs 0 2015-12-08 04:40
> /user/hive/warehouse/persontable/_SUCCESS
> -rw-r--r-- 1 hdfs hdfs 47 2015-12-08 04:40
> /user/hive/warehouse/persontable/part-00000
> -rw-r--r-- 1 hdfs hdfs 33 2015-12-08 04:40
> /user/hive/warehouse/persontable/part-00001
But when I try show tables in hive console ,I couldnt find the table.
> hive> use default ;
> OK
> Time taken: 0.864 seconds
> hive> show tables;
> OK
> dataframe_test
> sample_07
> sample_08
> Time taken: 0.521 seconds, Fetched: 3 row(s)
> hive> use xademo ;
> OK
> Time taken: 0.791 seconds
> hive> show tables;
> OK
> call_detail_records
> customer_details
> recharge_details
> Time taken: 0.256 seconds, Fetched: 3 row(s)
Can somebody guide me to right direction ,if something is wrong with the
code or I am unable to understand the concepts.
Would really appreciate your help.
Thanks,
Divya
Re: Unable to acces hive table (created through hive context) in hive console
Posted by Harsh J <ha...@cloudera.com>.
Are you certain you are providing Spark with the right Hive configuration?
Is there a valid HIVE_CONF_DIR defined in your spark-env.sh, with a
hive-site.xml detailing the location/etc. of the metastore service and/or
DB?
Without a valid metastore config, Hive may switch to using a local
(embedded) Derby DB and create metadata under that - thereby giving the
impression of things working, but not actually updating your true metastore
database.
You should be able to see your spark driver connect to your metastore if it
has been properly configured.
There's nothing incorrect with your program, it works just fine on my CDH5
Spark-on-YARN instance (and SHOW TABLES; inside Hive does show the
"persontable" created, after the program's completion).
On Tue, Dec 8, 2015 at 11:47 AM Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
>
> I am new bee to Spark and using HDP 2.2 which comes with Spark 1.3.1
> I tried following code example
>
>> import org.apache.spark.sql.SQLContext
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> import sqlContext.implicits._
>>
>> val personFile = "/user/hdfs/TestSpark/Person.csv"
>> val df = sqlContext.load(
>> "com.databricks.spark.csv",
>> Map("path" -> personFile, "header" -> "true", "inferSchema" ->
>> "true"))
>> df.printSchema()
>> val selectedData = df.select("Name", "Age")
>> selectedData.save("NewPerson.csv", "com.databricks.spark.csv")
>> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>> hiveContext.sql("CREATE TABLE IF NOT EXISTS PersonTable (Name STRING, Age
>> STRING)")
>> hiveContext.sql("LOAD DATA INPATH '/user/hdfs/NewPerson.csv' INTO TABLE
>> PersonTable")
>> hiveContext.sql("from PersonTable SELECT Name, Age
>> ").collect.foreach(println)
>
>
> I am able to access above table in HDFS
>
>> [hdfs@sandbox ~]$ hadoop fs -ls /user/hive/warehouse/persontable
>> Found 3 items
>> -rw-r--r-- 1 hdfs hdfs 0 2015-12-08 04:40
>> /user/hive/warehouse/persontable/_SUCCESS
>> -rw-r--r-- 1 hdfs hdfs 47 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00000
>> -rw-r--r-- 1 hdfs hdfs 33 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00001
>
>
> But when I try show tables in hive console ,I couldnt find the table.
>
>> hive> use default ;
>> OK
>> Time taken: 0.864 seconds
>> hive> show tables;
>> OK
>> dataframe_test
>> sample_07
>> sample_08
>> Time taken: 0.521 seconds, Fetched: 3 row(s)
>> hive> use xademo ;
>> OK
>> Time taken: 0.791 seconds
>> hive> show tables;
>> OK
>> call_detail_records
>> customer_details
>> recharge_details
>> Time taken: 0.256 seconds, Fetched: 3 row(s)
>
>
> Can somebody guide me to right direction ,if something is wrong with the
> code or I am unable to understand the concepts.
> Would really appreciate your help.
>
> Thanks,
> Divya
>
>
Re: Unable to acces hive table (created through hive context) in hive console
Posted by Harsh J <ha...@cloudera.com>.
Are you certain you are providing Spark with the right Hive configuration?
Is there a valid HIVE_CONF_DIR defined in your spark-env.sh, with a
hive-site.xml detailing the location/etc. of the metastore service and/or
DB?
Without a valid metastore config, Hive may switch to using a local
(embedded) Derby DB and create metadata under that - thereby giving the
impression of things working, but not actually updating your true metastore
database.
You should be able to see your spark driver connect to your metastore if it
has been properly configured.
There's nothing incorrect with your program, it works just fine on my CDH5
Spark-on-YARN instance (and SHOW TABLES; inside Hive does show the
"persontable" created, after the program's completion).
On Tue, Dec 8, 2015 at 11:47 AM Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
>
> I am new bee to Spark and using HDP 2.2 which comes with Spark 1.3.1
> I tried following code example
>
>> import org.apache.spark.sql.SQLContext
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> import sqlContext.implicits._
>>
>> val personFile = "/user/hdfs/TestSpark/Person.csv"
>> val df = sqlContext.load(
>> "com.databricks.spark.csv",
>> Map("path" -> personFile, "header" -> "true", "inferSchema" ->
>> "true"))
>> df.printSchema()
>> val selectedData = df.select("Name", "Age")
>> selectedData.save("NewPerson.csv", "com.databricks.spark.csv")
>> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>> hiveContext.sql("CREATE TABLE IF NOT EXISTS PersonTable (Name STRING, Age
>> STRING)")
>> hiveContext.sql("LOAD DATA INPATH '/user/hdfs/NewPerson.csv' INTO TABLE
>> PersonTable")
>> hiveContext.sql("from PersonTable SELECT Name, Age
>> ").collect.foreach(println)
>
>
> I am able to access above table in HDFS
>
>> [hdfs@sandbox ~]$ hadoop fs -ls /user/hive/warehouse/persontable
>> Found 3 items
>> -rw-r--r-- 1 hdfs hdfs 0 2015-12-08 04:40
>> /user/hive/warehouse/persontable/_SUCCESS
>> -rw-r--r-- 1 hdfs hdfs 47 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00000
>> -rw-r--r-- 1 hdfs hdfs 33 2015-12-08 04:40
>> /user/hive/warehouse/persontable/part-00001
>
>
> But when I try show tables in hive console ,I couldnt find the table.
>
>> hive> use default ;
>> OK
>> Time taken: 0.864 seconds
>> hive> show tables;
>> OK
>> dataframe_test
>> sample_07
>> sample_08
>> Time taken: 0.521 seconds, Fetched: 3 row(s)
>> hive> use xademo ;
>> OK
>> Time taken: 0.791 seconds
>> hive> show tables;
>> OK
>> call_detail_records
>> customer_details
>> recharge_details
>> Time taken: 0.256 seconds, Fetched: 3 row(s)
>
>
> Can somebody guide me to right direction ,if something is wrong with the
> code or I am unable to understand the concepts.
> Would really appreciate your help.
>
> Thanks,
> Divya
>
>