You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Marcelo Vanzin <va...@cloudera.com> on 2015/04/30 23:36:20 UTC
Uninitialized session in HiveContext?
Hey all,
We ran into some test failures in our internal branch (which builds
against Hive 1.1), and I narrowed it down to the fix below. I'm not
super familiar with the Hive integration code, but does this look like
a bug for other versions of Hive too?
This caused an error where some internal Hive configuration that is
initialized by the session were not available.
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
index dd06b26..6242745 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
@@ -93,6 +93,10 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) {
if (conf.dialect == "sql") {
super.sql(substituted)
} else if (conf.dialect == "hiveql") {
+ // Make sure Hive session state is initialized.
+ if (SessionState.get() != sessionState) {
+ SessionState.start(sessionState)
+ }
val ddlPlan = ddlParserWithHiveQL.parse(sqlText,
exceptionOnError = false)
DataFrame(this, ddlPlan.getOrElse(HiveQl.parseSql(substituted)))
} else {
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Uninitialized session in HiveContext?
Posted by Marcelo Vanzin <va...@cloudera.com>.
Hi Michael,
It would be great to see changes to make hive integration less
painful, and I can test them in our environment once you have a patch.
But I guess my question is a little more geared towards the current
code; doesn't the issue I ran into affect 1.4 and potentially earlier
versions too?
On Thu, Apr 30, 2015 at 5:19 PM, Michael Armbrust
<mi...@databricks.com> wrote:
> Hey Marcelo,
>
> Thanks for the heads up! I'm currently in the process of refactoring all of
> this (to separate the metadata connection from the execution side) and as
> part of this I'm making the initialization of the session not lazy. It
> would be great to hear if this also works for your internal integration
> tests once the patch is up (hopefully this weekend).
>
> Michael
>
> On Thu, Apr 30, 2015 at 2:36 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> Hey all,
>>
>> We ran into some test failures in our internal branch (which builds
>> against Hive 1.1), and I narrowed it down to the fix below. I'm not
>> super familiar with the Hive integration code, but does this look like
>> a bug for other versions of Hive too?
>>
>> This caused an error where some internal Hive configuration that is
>> initialized by the session were not available.
>>
>> diff --git
>> a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
>> b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
>> index dd06b26..6242745 100644
>> --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
>> +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
>> @@ -93,6 +93,10 @@ class HiveContext(sc: SparkContext) extends
>> SQLContext(sc) {
>> if (conf.dialect == "sql") {
>> super.sql(substituted)
>> } else if (conf.dialect == "hiveql") {
>> + // Make sure Hive session state is initialized.
>> + if (SessionState.get() != sessionState) {
>> + SessionState.start(sessionState)
>> + }
>> val ddlPlan = ddlParserWithHiveQL.parse(sqlText,
>> exceptionOnError = false)
>> DataFrame(this, ddlPlan.getOrElse(HiveQl.parseSql(substituted)))
>> } else {
>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Uninitialized session in HiveContext?
Posted by Michael Armbrust <mi...@databricks.com>.
Hey Marcelo,
Thanks for the heads up! I'm currently in the process of refactoring all
of this (to separate the metadata connection from the execution side) and
as part of this I'm making the initialization of the session not lazy. It
would be great to hear if this also works for your internal integration
tests once the patch is up (hopefully this weekend).
Michael
On Thu, Apr 30, 2015 at 2:36 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hey all,
>
> We ran into some test failures in our internal branch (which builds
> against Hive 1.1), and I narrowed it down to the fix below. I'm not
> super familiar with the Hive integration code, but does this look like
> a bug for other versions of Hive too?
>
> This caused an error where some internal Hive configuration that is
> initialized by the session were not available.
>
> diff --git
> a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
> b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
> index dd06b26..6242745 100644
> --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
> +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
> @@ -93,6 +93,10 @@ class HiveContext(sc: SparkContext) extends
> SQLContext(sc) {
> if (conf.dialect == "sql") {
> super.sql(substituted)
> } else if (conf.dialect == "hiveql") {
> + // Make sure Hive session state is initialized.
> + if (SessionState.get() != sessionState) {
> + SessionState.start(sessionState)
> + }
> val ddlPlan = ddlParserWithHiveQL.parse(sqlText,
> exceptionOnError = false)
> DataFrame(this, ddlPlan.getOrElse(HiveQl.parseSql(substituted)))
> } else {
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>