You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Christopher Penney <cp...@gmail.com> on 2013/09/03 15:04:11 UTC
Hive with Kerberos and a Remote Metastore
I'm new to hive and trying to set it up in a relatively secure manner for a
test environment. I want to use a remote metastore so MR jobs can access
the DB. I seem to have things almost working, but when a user with a
credential tries to create a database I get:
hive> show databases;
OK
default
hive> create database testdb;
FAILED: Error in metadata: MetaException(message:Got exception:
org.apache.hadoop.ipc.RemoteException User: hdfs/
hadoopserver.sub.dom.com@SUB.DOM.COM is not allowed to impersonate
myuserid@SUB.DOM.COM)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
I have "hive --service metastore" running as hdfs with hdfs/
hadoopserver.sub.dom.com@SUB.DOM.COM as the principal. I'm running hive as
"myuserid" on the same box. I don't know if it's related, but if I try to
run hive from another system I get a GSS Initiate error unless I use the
same principal (hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM) for
hive.metastore.kerberos.principal. Is that expected?
When I try googling this I see similar issues, but the message about not
being able to impersonate only shows the single part user name where for me
it's showing the realm. I tried playing with the auth_to_local property,
but it didn't help. Map Reduce and HDFS operations are working fine
otherwise.
In core-site.xml I have:
<property>
<name>hadoop.proxyuser.hdfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
In hive-site.xml I have:
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore</value>
<description>the URL of the MySQL database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoopserver.sub.dom.com:9083</value>
</property>
<property>
<name>hive.security.authorization.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.sasl.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.kerberos.keytab.file</name>
<value>/etc/hadoop/hdfs.keytab</value>
</property>
<property>
<name>hive.metastore.kerberos.principal</name>
<value>hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
Any ideas?
Re: Hive with Kerberos and a Remote Metastore
Posted by Subroto <ss...@datameer.com>.
Hi Christopher,
I am running hive metastore as user "hive" (hive/ip-10-151-109-165.ec2.internal@EC2.INTERNAL) and then I configure hadoop.proxyuser.hive.hosts and hadoop.proxyuser.hive.groups to '*'.
This works.
On Sep 3, 2013, at 6:39 PM, Subroto wrote:
> I am also facing the same problem…. Any idea??
>
> Cheers,
> Subroto Sanyal
> On Sep 3, 2013, at 3:04 PM, Christopher Penney wrote:
>
>> I'm new to hive and trying to set it up in a relatively secure manner for a test environment. I want to use a remote metastore so MR jobs can access the DB. I seem to have things almost working, but when a user with a credential tries to create a database I get:
>>
>> hive> show databases;
>> OK
>> default
>> hive> create database testdb;
>> FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.ipc.RemoteException User: hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM is not allowed to impersonate myuserid@SUB.DOM.COM)
>> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>>
>> I have "hive --service metastore" running as hdfs with hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM as the principal. I'm running hive as "myuserid" on the same box. I don't know if it's related, but if I try to run hive from another system I get a GSS Initiate error unless I use the same principal (hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM) for hive.metastore.kerberos.principal. Is that expected?
>>
>> When I try googling this I see similar issues, but the message about not being able to impersonate only shows the single part user name where for me it's showing the realm. I tried playing with the auth_to_local property, but it didn't help. Map Reduce and HDFS operations are working fine otherwise.
>>
>> In core-site.xml I have:
>>
>> <property>
>> <name>hadoop.proxyuser.hdfs.hosts</name>
>> <value>*</value>
>> </property>
>>
>> <property>
>> <name>hadoop.proxyuser.hdfs.groups</name>
>> <value>*</value>
>> </property>
>>
>> In hive-site.xml I have:
>>
>> <property>
>> <name>javax.jdo.option.ConnectionURL</name>
>> <value>jdbc:mysql://localhost/metastore</value>
>> <description>the URL of the MySQL database</description>
>> </property>
>>
>> <property>
>> <name>javax.jdo.option.ConnectionDriverName</name>
>> <value>com.mysql.jdbc.Driver</value>
>> </property>
>>
>> <property>
>> <name>javax.jdo.option.ConnectionUserName</name>
>> <value>hive</value>
>> </property>
>>
>> <property>
>> <name>javax.jdo.option.ConnectionPassword</name>
>> <value>password</value>
>> </property>
>>
>> <property>
>> <name>datanucleus.autoCreateSchema</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>datanucleus.fixedDatastore</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>hive.metastore.uris</name>
>> <value>thrift://hadoopserver.sub.dom.com:9083</value>
>> </property>
>>
>> <property>
>> <name>hive.security.authorization.enabled</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>hive.metastore.sasl.enabled</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>hive.metastore.kerberos.keytab.file</name>
>> <value>/etc/hadoop/hdfs.keytab</value>
>> </property>
>>
>> <property>
>> <name>hive.metastore.kerberos.principal</name>
>> <value>hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM</value>
>> </property>
>>
>> <property>
>> <name>hive.metastore.execute.setugi</name>
>> <value>true</value>
>> </property>
>>
>> Any ideas?
>>
>
Re: Hive with Kerberos and a Remote Metastore
Posted by Subroto <ss...@datameer.com>.
I am also facing the same problem…. Any idea??
Cheers,
Subroto Sanyal
On Sep 3, 2013, at 3:04 PM, Christopher Penney wrote:
> I'm new to hive and trying to set it up in a relatively secure manner for a test environment. I want to use a remote metastore so MR jobs can access the DB. I seem to have things almost working, but when a user with a credential tries to create a database I get:
>
> hive> show databases;
> OK
> default
> hive> create database testdb;
> FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.ipc.RemoteException User: hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM is not allowed to impersonate myuserid@SUB.DOM.COM)
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>
> I have "hive --service metastore" running as hdfs with hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM as the principal. I'm running hive as "myuserid" on the same box. I don't know if it's related, but if I try to run hive from another system I get a GSS Initiate error unless I use the same principal (hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM) for hive.metastore.kerberos.principal. Is that expected?
>
> When I try googling this I see similar issues, but the message about not being able to impersonate only shows the single part user name where for me it's showing the realm. I tried playing with the auth_to_local property, but it didn't help. Map Reduce and HDFS operations are working fine otherwise.
>
> In core-site.xml I have:
>
> <property>
> <name>hadoop.proxyuser.hdfs.hosts</name>
> <value>*</value>
> </property>
>
> <property>
> <name>hadoop.proxyuser.hdfs.groups</name>
> <value>*</value>
> </property>
>
> In hive-site.xml I have:
>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/metastore</value>
> <description>the URL of the MySQL database</description>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>password</value>
> </property>
>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.uris</name>
> <value>thrift://hadoopserver.sub.dom.com:9083</value>
> </property>
>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.sasl.enabled</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.metastore.kerberos.keytab.file</name>
> <value>/etc/hadoop/hdfs.keytab</value>
> </property>
>
> <property>
> <name>hive.metastore.kerberos.principal</name>
> <value>hdfs/hadoopserver.sub.dom.com@SUB.DOM.COM</value>
> </property>
>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> </property>
>
> Any ideas?
>