You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Terry Hole <hu...@gmail.com> on 2015/07/10 05:59:48 UTC
[Spark Hive SQL] Set the hive connection in hive context is broken in
spark 1.4.1-rc1?
Hi,
I am trying to set the hive metadata destination to a mysql database in
hive context, it works fine in spark 1.3.1, but it seems broken in spark
1.4.1-rc1, where it always connect to the default metadata: local), is this
a regression or we must set the connection in hive-site.xml?
The code is very simple in spark shell:
* import org.apache.spark.sql.hive._*
* val hiveContext = new HiveContext(sc)*
* hiveContext.setConf("javax.jdo.option.ConnectionDriveName",
"com.mysql.jdbc.Driver")*
* hiveContext.setConf("javax.jdo.option.ConnectionUserName", "hive")*
* hiveContext.setConf("javax.jdo.option.ConnectionPassword", "hive")*
* hiveContext.setConf("javax.jdo.option.ConnectionURL",
"jdbc:mysql://10.111.3.186:3306/hive <http://10.111.3.186:3306/hive>")*
* hiveContext.setConf("hive.metastore.warehouse.dir",
"/user/hive/warehouse")*
* hiveContext.sql("select * from mysqltable").show()*
*Thanks!*
*-Terry*
Re: [Spark Hive SQL] Set the hive connection in hive context is
broken in spark 1.4.1-rc1?
Posted by Terry Hole <hu...@gmail.com>.
Michael,
Thanks
- Terry
Michael Armbrust <mi...@databricks.com>于2015年7月11日星期六 04:02写道：
> Metastore configuration should be set in hive-site.xml.
>
> On Thu, Jul 9, 2015 at 8:59 PM, Terry Hole <hu...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to set the hive metadata destination to a mysql database in
>> hive context, it works fine in spark 1.3.1, but it seems broken in spark
>> 1.4.1-rc1, where it always connect to the default metadata: local), is this
>> a regression or we must set the connection in hive-site.xml?
>>
>> The code is very simple in spark shell:
>> * import org.apache.spark.sql.hive._*
>> * val hiveContext = new HiveContext(sc)*
>> * hiveContext.setConf("javax.jdo.option.ConnectionDriveName",
>> "com.mysql.jdbc.Driver")*
>> * hiveContext.setConf("javax.jdo.option.ConnectionUserName", "hive")*
>> * hiveContext.setConf("javax.jdo.option.ConnectionPassword", "hive")*
>> * hiveContext.setConf("javax.jdo.option.ConnectionURL",
>> "jdbc:mysql://10.111.3.186:3306/hive <http://10.111.3.186:3306/hive>")*
>> * hiveContext.setConf("hive.metastore.warehouse.dir",
>> "/user/hive/warehouse")*
>> * hiveContext.sql("select * from mysqltable").show()*
>>
>> *Thanks!*
>> *-Terry*
>>
>>
>
Re: [Spark Hive SQL] Set the hive connection in hive context is
broken in spark 1.4.1-rc1?
Posted by Michael Armbrust <mi...@databricks.com>.
Metastore configuration should be set in hive-site.xml.
On Thu, Jul 9, 2015 at 8:59 PM, Terry Hole <hu...@gmail.com> wrote:
> Hi,
>
> I am trying to set the hive metadata destination to a mysql database in
> hive context, it works fine in spark 1.3.1, but it seems broken in spark
> 1.4.1-rc1, where it always connect to the default metadata: local), is this
> a regression or we must set the connection in hive-site.xml?
>
> The code is very simple in spark shell:
> * import org.apache.spark.sql.hive._*
> * val hiveContext = new HiveContext(sc)*
> * hiveContext.setConf("javax.jdo.option.ConnectionDriveName",
> "com.mysql.jdbc.Driver")*
> * hiveContext.setConf("javax.jdo.option.ConnectionUserName", "hive")*
> * hiveContext.setConf("javax.jdo.option.ConnectionPassword", "hive")*
> * hiveContext.setConf("javax.jdo.option.ConnectionURL",
> "jdbc:mysql://10.111.3.186:3306/hive <http://10.111.3.186:3306/hive>")*
> * hiveContext.setConf("hive.metastore.warehouse.dir",
> "/user/hive/warehouse")*
> * hiveContext.sql("select * from mysqltable").show()*
>
> *Thanks!*
> *-Terry*
>
>