You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jay Singh <ja...@apttus.com> on 2019/02/21 07:43:23 UTC

Spark-hive integration on HDInsight

I am trying to integrate  spark with hive on HDInsight  spark cluster .
I copied hive-site.xml in spark/conf directory. In addition I added hive metastore properties like jdbc connection info on Ambari as well. But still the database and tables created using spark-sql are not visible in hive. Changed 'spark.sql.warehouse.dir' value also to point to hive warehouse directory.
Spark does work with hive not having LLAP ON. What am I missing in the configuration to integrate spark with hive ? Any pointer will be appreciated.

thx

Re: Spark-hive integration on HDInsight

Posted by amit kumar singh <am...@gmail.com>.
Hey jay

How you are making your cluster  are you using spark cluster 

All this thing should be set up automatically 



Sent from my iPhone

> On Feb 21, 2019, at 12:12 PM, Felix Cheung <fe...@hotmail.com> wrote:
> 
> You should check with HDInsight support
> 
> From: Jay Singh <ja...@apttus.com>
> Sent: Wednesday, February 20, 2019 11:43:23 PM
> To: User
> Subject: Spark-hive integration on HDInsight
>  
> I am trying to integrate  spark with hive on HDInsight  spark cluster .
> I copied hive-site.xml in spark/conf directory. In addition I added hive metastore properties like jdbc connection info on Ambari as well. But still the database and tables created using spark-sql are not visible in hive. Changed ‘spark.sql.warehouse.dir’ value also to point to hive warehouse directory.
> Spark does work with hive not having LLAP ON. What am I missing in the configuration to integrate spark with hive ? Any pointer will be appreciated.
>  
> thx

Re: Spark-hive integration on HDInsight

Posted by Felix Cheung <fe...@hotmail.com>.
You should check with HDInsight support

________________________________
From: Jay Singh <ja...@apttus.com>
Sent: Wednesday, February 20, 2019 11:43:23 PM
To: User
Subject: Spark-hive integration on HDInsight

I am trying to integrate  spark with hive on HDInsight  spark cluster .
I copied hive-site.xml in spark/conf directory. In addition I added hive metastore properties like jdbc connection info on Ambari as well. But still the database and tables created using spark-sql are not visible in hive. Changed ‘spark.sql.warehouse.dir’ value also to point to hive warehouse directory.
Spark does work with hive not having LLAP ON. What am I missing in the configuration to integrate spark with hive ? Any pointer will be appreciated.

thx