You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Stephen Boesch <ja...@gmail.com> on 2012/11/24 23:19:44 UTC
Custom hive-site.xml is ignored, how to find out why
It seems the customized hive-site.xml is not being read. It lives under
$HIVE_HOME/conf ( which happens to be /shared/hive/conf). I have tried
everything there is to try: set HIVE_CONF_DIR=/shared/hive/conf , added
--config /shared/hive/conf and added debugging to the hive shell script
(bash -x) to ensure that the correct directory is actually getting used
The properties inside hive-site.xml are getting ignored: originally set to
using mysql , but instead the default derby is getting used. Then tried
changing hive.metastore.local between true and false: no difference in
behavior - just going to /tmp/$USER for creating the derby no matter what.
I wondered whether hive-site.xml maybe has a syntax error and were getting
ignored: so removed everything except <configuration></configuration>:
still no
This is the totally simplified hive-site.xml: it just has enough to try to
see if it is actually being read/applied (and it is not..):
steve@mithril:/shared/hive/conf$ cat hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>/shared/hive/lib/hive-hwi-0.9.0.war</value>
<description>This is the WAR file with the jsp content for Hive Web
Interface</description>
</property>
</configuration>
After running some DDL in hive, for example, no files are created
underneath /tmp/hive (instead they are going to /tmp/$USER which is
default - as if the custom hive-site.xml never existed.
Re: Custom hive-site.xml is ignored, how to find out why
Posted by Stephen Boesch <ja...@gmail.com>.
Hi,
The problem may not have to do with hive-site.xml. When I run hive
client by itself it connects successfully to mysql and creates / reads
metadataa.
The problem comes in when i run the metastore/thrift servers via: hive
--service metastore hive --service hiveserver. As soon as i do that,
and try to run hive client (hive cli) then it goes back to trying to use
derby.
There must be additional settings required for the hive metastore service
beyond those in my hive-site.xml.
Btw from OP i aleady have the settings in hive-site.xml for connecting via
jdo to mysql. I don't have the data nucleus ones. Are those required?
The docs i saw did not mention them.
2012/11/26 Shreepadma Venugopalan <sh...@cloudera.com>
> Hi Stephen,
>
> If you wish to setup a mysql metastore, you need to have the following in
> your hive-site.xml,
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://MYHOST/metastore</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hiveuser</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>password</value>
> </property>
>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>true</value>
> </property>
>
>
> Thanks.
> Shreepadma
>
>
> On Sat, Nov 24, 2012 at 8:41 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
>> It appears that I were missing the *hive.metastore.uris* parameter.
>> That one was not mentioned in the (several) blogs / tutorials that I had
>> seen.
>>
>>
>> 2012/11/24 Stephen Boesch <ja...@gmail.com>
>>
>>>
>>> It seems the customized hive-site.xml is not being read. It lives under
>>> $HIVE_HOME/conf ( which happens to be /shared/hive/conf). I have tried
>>> everything there is to try: set HIVE_CONF_DIR=/shared/hive/conf , added
>>> --config /shared/hive/conf and added debugging to the hive shell script
>>> (bash -x) to ensure that the correct directory is actually getting used
>>>
>>> The properties inside hive-site.xml are getting ignored: originally set
>>> to using mysql , but instead the default derby is getting used. Then tried
>>> changing hive.metastore.local between true and false: no difference in
>>> behavior - just going to /tmp/$USER for creating the derby no matter what.
>>>
>>> I wondered whether hive-site.xml maybe has a syntax error and were
>>> getting ignored: so removed everything except
>>> <configuration></configuration>: still no
>>>
>>> This is the totally simplified hive-site.xml: it just has enough to try
>>> to see if it is actually being read/applied (and it is not..):
>>>
>>>
>>> steve@mithril:/shared/hive/conf$ cat hive-site.xml
>>> <?xml version="1.0"?>
>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>
>>> <configuration>
>>>
>>> <!-- Hive Execution Parameters -->
>>> <property>
>>> <name>hive.metastore.local</name>
>>> <value>false</value>
>>> </property>
>>>
>>> <property>
>>> <name>hive.exec.scratchdir</name>
>>> <value>/tmp/hive/hive-${user.name}</value>
>>> <description>Scratch space for Hive jobs</description>
>>> </property>
>>>
>>> <property>
>>> <name>hive.hwi.war.file</name>
>>> <value>/shared/hive/lib/hive-hwi-0.9.0.war</value>
>>> <description>This is the WAR file with the jsp content for Hive Web
>>> Interface</description>
>>> </property>
>>>
>>> </configuration>
>>>
>>>
>>> After running some DDL in hive, for example, no files are created
>>> underneath /tmp/hive (instead they are going to /tmp/$USER which is
>>> default - as if the custom hive-site.xml never existed.
>>>
>>>
>>>
>>
>
Re: Custom hive-site.xml is ignored, how to find out why
Posted by Shreepadma Venugopalan <sh...@cloudera.com>.
Hi Stephen,
If you wish to setup a mysql metastore, you need to have the following in
your hive-site.xml,
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://MYHOST/metastore</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
Thanks.
Shreepadma
On Sat, Nov 24, 2012 at 8:41 PM, Stephen Boesch <ja...@gmail.com> wrote:
> It appears that I were missing the *hive.metastore.uris* parameter. That
> one was not mentioned in the (several) blogs / tutorials that I had seen.
>
>
> 2012/11/24 Stephen Boesch <ja...@gmail.com>
>
>>
>> It seems the customized hive-site.xml is not being read. It lives under
>> $HIVE_HOME/conf ( which happens to be /shared/hive/conf). I have tried
>> everything there is to try: set HIVE_CONF_DIR=/shared/hive/conf , added
>> --config /shared/hive/conf and added debugging to the hive shell script
>> (bash -x) to ensure that the correct directory is actually getting used
>>
>> The properties inside hive-site.xml are getting ignored: originally set
>> to using mysql , but instead the default derby is getting used. Then tried
>> changing hive.metastore.local between true and false: no difference in
>> behavior - just going to /tmp/$USER for creating the derby no matter what.
>>
>> I wondered whether hive-site.xml maybe has a syntax error and were
>> getting ignored: so removed everything except
>> <configuration></configuration>: still no
>>
>> This is the totally simplified hive-site.xml: it just has enough to try
>> to see if it is actually being read/applied (and it is not..):
>>
>>
>> steve@mithril:/shared/hive/conf$ cat hive-site.xml
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <configuration>
>>
>> <!-- Hive Execution Parameters -->
>> <property>
>> <name>hive.metastore.local</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>hive.exec.scratchdir</name>
>> <value>/tmp/hive/hive-${user.name}</value>
>> <description>Scratch space for Hive jobs</description>
>> </property>
>>
>> <property>
>> <name>hive.hwi.war.file</name>
>> <value>/shared/hive/lib/hive-hwi-0.9.0.war</value>
>> <description>This is the WAR file with the jsp content for Hive Web
>> Interface</description>
>> </property>
>>
>> </configuration>
>>
>>
>> After running some DDL in hive, for example, no files are created
>> underneath /tmp/hive (instead they are going to /tmp/$USER which is
>> default - as if the custom hive-site.xml never existed.
>>
>>
>>
>
Re: Custom hive-site.xml is ignored, how to find out why
Posted by Stephen Boesch <ja...@gmail.com>.
It appears that I were missing the *hive.metastore.uris* parameter. That
one was not mentioned in the (several) blogs / tutorials that I had seen.
2012/11/24 Stephen Boesch <ja...@gmail.com>
>
> It seems the customized hive-site.xml is not being read. It lives under
> $HIVE_HOME/conf ( which happens to be /shared/hive/conf). I have tried
> everything there is to try: set HIVE_CONF_DIR=/shared/hive/conf , added
> --config /shared/hive/conf and added debugging to the hive shell script
> (bash -x) to ensure that the correct directory is actually getting used
>
> The properties inside hive-site.xml are getting ignored: originally set to
> using mysql , but instead the default derby is getting used. Then tried
> changing hive.metastore.local between true and false: no difference in
> behavior - just going to /tmp/$USER for creating the derby no matter what.
>
> I wondered whether hive-site.xml maybe has a syntax error and were getting
> ignored: so removed everything except <configuration></configuration>:
> still no
>
> This is the totally simplified hive-site.xml: it just has enough to try to
> see if it is actually being read/applied (and it is not..):
>
>
> steve@mithril:/shared/hive/conf$ cat hive-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <configuration>
>
> <!-- Hive Execution Parameters -->
> <property>
> <name>hive.metastore.local</name>
> <value>false</value>
> </property>
>
> <property>
> <name>hive.exec.scratchdir</name>
> <value>/tmp/hive/hive-${user.name}</value>
> <description>Scratch space for Hive jobs</description>
> </property>
>
> <property>
> <name>hive.hwi.war.file</name>
> <value>/shared/hive/lib/hive-hwi-0.9.0.war</value>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>
> </property>
>
> </configuration>
>
>
> After running some DDL in hive, for example, no files are created
> underneath /tmp/hive (instead they are going to /tmp/$USER which is
> default - as if the custom hive-site.xml never existed.
>
>
>