You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Changshu Liu (JIRA)" <ji...@apache.org> on 2018/04/09 02:45:00 UTC
[jira] [Comment Edited] (HIVE-13532) MapredLocalTask should use the
same security settings as remote task
[ https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429881#comment-16429881 ]
Changshu Liu edited comment on HIVE-13532 at 4/9/18 2:44 AM:
-------------------------------------------------------------
We encountered this issue using Hive 1.2 branch, the problem is that the MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security settings.
We just need to pass the correct Hadoop permission information down to local task via env variable.
here are the full call stack if this issue happened:
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator (Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator (Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask (MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local work failed
java.io.IOException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: abc is not allowed to impersonate abc
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
was (Author: csliu):
We encountered this issue using Hive 1.2 branch, the problem is that the MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security settings, here are the full call stack:
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator (Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator (Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask (MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local work failed
java.io.IOException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: abc is not allowed to impersonate abc
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> MapredLocalTask should use the same security settings as remote task
> --------------------------------------------------------------------
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
> Reporter: Zhiwen Sun
> Assignee: Changshu Liu
> Priority: Major
> Attachments: HIVE-13532.1.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
> String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: abc is not allowed to impersonate
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
> String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)