You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Afshartous, Nick" <na...@turbine.com> on 2016/03/03 14:20:00 UTC
Using Spark SQL / Hive on AWS EMR
Hi,
On AWS EMR 4.2 / Spark 1.5.2, I tried the example here
https://spark.apache.org/docs/1.5.0/sql-programming-guide.html#hive-tables
to load data from a file into a Hive table.
scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'data.txt' INTO TABLE src")
The resultant error is below. Just wondering if I'm missing any steps in getting Hive setup on the AWS EMR Spark setup.
Thanks,
--
Nick
16/03/02 14:14:04 INFO Hive: Renaming src: file:/home/hadoop/data.txt, dest: hdfs://ip-10-247-128-59.ec2.internal:8020/user/hive/warehouse/src/data_copy_2.txt, Status:true
16/03/02 14:14:04 WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.TApplicationException: Invalid method name: 'alter_table_with_cascade'
Re: Using Spark SQL / Hive on AWS EMR
Posted by Suresh Thalamati <su...@gmail.com>.
the error ('Invalid method name: ‘alter_table_with_cascade’’) you are seeing may be related to mismatch of hive versions. Error looks similar to one reported in https://issues.apache.org/jira/browse/SPARK-12496
> On Mar 3, 2016, at 7:43 AM, Gourav Sengupta <go...@gmail.com> wrote:
>
> Hi,
>
> Why are you trying to load data into HIVE and then access it via hiveContext? (by the way hiveContext tables are not visible in the sqlContext).
>
> Please read the data directly into a SPARK dataframe and then register it as a temp table to run queries on it.
>
>
> Regards,
> Gourav
>
> On Thu, Mar 3, 2016 at 1:20 PM, Afshartous, Nick <nafshartous@turbine.com <ma...@turbine.com>> wrote:
>
> Hi,
>
>
> On AWS EMR 4.2 / Spark 1.5.2, I tried the example here
>
> https://spark.apache.org/docs/1.5.0/sql-programming-guide.html#hive-tables <https://spark.apache.org/docs/1.5.0/sql-programming-guide.html#hive-tables>
>
> to load data from a file into a Hive table.
>
>
> scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>
> scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'data.txt' INTO TABLE src")
>
>
>
> The resultant error is below. Just wondering if I'm missing any steps in getting Hive setup on the AWS EMR Spark setup.
>
>
> Thanks,
> --
>
> Nick
>
>
>
> 16/03/02 14:14:04 INFO Hive: Renaming src: file:/home/hadoop/data.txt, dest: hdfs://ip-10-247-128-59.ec2.internal:8020/user/hive/warehouse/src/data_copy_2.txt, Status:true
> 16/03/02 14:14:04 WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.
> org.apache.thrift.TApplicationException: Invalid method name: 'alter_table_with_cascade'
>
Re: Using Spark SQL / Hive on AWS EMR
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
Why are you trying to load data into HIVE and then access it via
hiveContext? (by the way hiveContext tables are not visible in the
sqlContext).
Please read the data directly into a SPARK dataframe and then register it
as a temp table to run queries on it.
Regards,
Gourav
On Thu, Mar 3, 2016 at 1:20 PM, Afshartous, Nick <na...@turbine.com>
wrote:
>
> Hi,
>
>
> On AWS EMR 4.2 / Spark 1.5.2, I tried the example here
>
>
>
> https://spark.apache.org/docs/1.5.0/sql-programming-guide.html#hive-tables
>
>
> to load data from a file into a Hive table.
>
>
> scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
> STRING)")
>
> scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'data.txt' INTO TABLE src")
>
>
>
> The resultant error is below. Just wondering if I'm missing any steps in
> getting Hive setup on the AWS EMR Spark setup.
>
>
> Thanks,
>
> --
>
> Nick
>
>
>
> 16/03/02 14:14:04 INFO Hive: Renaming src: file:/home/hadoop/data.txt,
> dest:
> hdfs://ip-10-247-128-59.ec2.internal:8020/user/hive/warehouse/src/data_copy_2.txt,
> Status:true
> 16/03/02 14:14:04 WARN RetryingMetaStoreClient: MetaStoreClient lost
> connection. Attempting to reconnect.
> org.apache.thrift.TApplicationException: Invalid method name:
> 'alter_table_with_cascade'
>