You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by hnagar <he...@mobiusws.com> on 2016/01/19 20:32:18 UTC
Spark SQL -Hive transactions support
Hive has transactions support since version 0.14.
I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark
SQL now. I tried in the Spark-Shell and it gives the following error
org.apache.spark.sql.AnalysisException:
Unsupported language features in query: insert into test values(1, 'john',
'dong')
I am wondering if its a configuration issue, or this is still not supported
in Spark.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark SQL -Hive transactions support
Posted by Elliot West <te...@gmail.com>.
Hive's ACID feature (which introduces transactions) is not required for
inserts, only updates and deletes. Inserts should be supported on a vanilla
Hive shell. I'm not sure how Spark interacts with Hive in that regard but
perhaps the HiveSQLContext implementation is lacking support.
On a separate note, ACID is rarely configured out of the box and also
requires that your tables are declared with some specific attributes. This
might be worth checking but does not seem like the issue in your case:
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
For streaming type processing it would be nice to integrate Spark with
Hive's streaming API which uses the ACID feature:
https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
Additionally, updates and deletes could be supported by Spark by
integrating with the not yet released mutation API:
http://htmlpreview.github.io/?https://github.com/apache/hive/blob/master/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/package.html
Elliot.
On Tuesday, 19 January 2016, hnagar <he...@mobiusws.com> wrote:
> Hive has transactions support since version 0.14.
>
> I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark
> SQL now. I tried in the Spark-Shell and it gives the following error
>
> org.apache.spark.sql.AnalysisException:
> Unsupported language features in query: insert into test values(1, 'john',
> 'dong')
>
> I am wondering if its a configuration issue, or this is still not supported
> in Spark.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
RE: Spark SQL -Hive transactions support
Posted by Hemang Nagar <he...@mobiusws.com>.
Do you have any plans for supporting hive transactions in Spark?
From: Michael Armbrust [mailto:michael@databricks.com]
Sent: Tuesday, January 19, 2016 3:18 PM
To: hnagar <he...@mobiusws.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Spark SQL -Hive transactions support
We don't support Hive style transaction.
On Tue, Jan 19, 2016 at 11:32 AM, hnagar <hemang.nagar@mobiusws.com <ma...@mobiusws.com> > wrote:
Hive has transactions support since version 0.14.
I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark
SQL now. I tried in the Spark-Shell and it gives the following error
org.apache.spark.sql.AnalysisException:
Unsupported language features in query: insert into test values(1, 'john',
'dong')
I am wondering if its a configuration issue, or this is still not supported
in Spark.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
Re: Spark SQL -Hive transactions support
Posted by Michael Armbrust <mi...@databricks.com>.
We don't support Hive style transaction.
On Tue, Jan 19, 2016 at 11:32 AM, hnagar <he...@mobiusws.com> wrote:
> Hive has transactions support since version 0.14.
>
> I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark
> SQL now. I tried in the Spark-Shell and it gives the following error
>
> org.apache.spark.sql.AnalysisException:
> Unsupported language features in query: insert into test values(1, 'john',
> 'dong')
>
> I am wondering if its a configuration issue, or this is still not supported
> in Spark.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>