You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Sushanth Sowmyan (JIRA)" <ji...@apache.org> on 2014/03/24 22:50:59 UTC
[jira] [Commented] (HIVE-5523) HiveHBaseStorageHandler should pass
kerbros credentials down to HBase
[ https://issues.apache.org/jira/browse/HIVE-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945752#comment-13945752 ]
Sushanth Sowmyan commented on HIVE-5523:
----------------------------------------
Per my testing, this is actually working currently. From code-reading, I was initially convinced there was a bug, but it does seem to properly initialize hbase delegation tokens for read and write jobs, from hive and hcatalog, but it currently works for me.
I ran the following:
To create the table:
{noformat}
CREATE TABLE sushhbdata(rowkey STRING, pageviews STRING, bytes STRING);
LOAD DATA LOCAL INPATH 'sushhbdata' INTO TABLE sushhbdata;
CREATE TABLE IF NOT EXISTS sushpagecounts_hbase (rowkey STRING, pageviews STRING, bytes STRING)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ('hbase.columns.mapping' = ':key,f:c1,f:c2')
TBLPROPERTIES ('hbase.table.name' = 'sushpagecounts');
{noformat}
To insert data into the table:
{noformat}
set hive.execution.engine=mr;
FROM sushhbdata INSERT INTO TABLE sushpagecounts_hbase SELECT * where rowkey like 'EN%';
{noformat}
For the actual data, I just randomly generated strings of varying length in all caps, there were quite a few that began with "EN".
(Note that I set the execution engine to mr because there is another unrelated tez issue that blocks it currently if you don't set it, and I think Gunther or Sergei are trying to fix that. And I don't need aux jars either for our current hive commandline.)
I have a patch I'd suggest up for this anyway to make life a little easier for those who read this section of code in the future.
Could you please verify?
> HiveHBaseStorageHandler should pass kerbros credentials down to HBase
> ---------------------------------------------------------------------
>
> Key: HIVE-5523
> URL: https://issues.apache.org/jira/browse/HIVE-5523
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
> Affects Versions: 0.11.0
> Reporter: Nick Dimiduk
> Attachments: Task Logs_ 'attempt_201310110032_0023_r_000000_0'.html
>
>
> Running on a secured cluster, I have an HBase table defined thusly
> {noformat}
> CREATE TABLE IF NOT EXISTS pagecounts_hbase (rowkey STRING, pageviews STRING, bytes STRING)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ('hbase.columns.mapping' = ':key,f:c1,f:c2')
> TBLPROPERTIES ('hbase.table.name' = 'pagecounts');
> {noformat}
> and a query to populate that table
> {noformat}
> -- ensure hbase dependency jars are shipped with the MR job
> SET hive.aux.jars.path = file:///etc/hbase/conf/hbase-site.xml,file:///usr/lib/hive/lib/hive-hbase-handler-0.11.0.1.3.2.0-111.jar,file:///usr/lib/hbase/hbase-0.94.6.1.3.2.0-111-security.jar,file:///usr/lib/zookeeper/zookeeper-3.4.5.1.3.2.0-111.jar;
> -- populate our hbase table
> FROM pgc INSERT INTO TABLE pagecounts_hbase SELECT pgc.* WHERE rowkey LIKE 'en/q%' LIMIT 10;
> {noformat}
> The reduce tasks fail with what boils down to the following exception:
> {noformat}
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:263)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:590)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:224)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:313)
> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
> at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:104)
> at $Proxy10.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:146)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1346)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1305)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1292)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1001)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:896)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:998)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:900)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:857)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat.getHiveRecordWriter(HiveHBaseTableOutputFormat.java:83)
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:250)
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)