You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Loïc Chanel <lo...@telecomnancy.net> on 2015/07/23 12:10:51 UTC
Hive server identity assertion
Hi all,
As I am now exploring how Ranger works with Hive, I made some policies, but
it seems that group policies are not enforced.
Therefore, I was wondering how the Ranger plugin running on Hive was
asserting the user's identity.
I am even more surprised by the fact that I do not have any problem with
Ranger plugin working on HDFS, which is running on the exact same node.
In parallel, I know that Know plugin, for example, runs in a totally
different way, but as it seems that, as does HBase, Hive does not provide
with any user mapping function, I thought the identity would be asserted on
the node Hive Server is running on, as if the user was a Unix one.
Do someone as an idea about how the user groups can be founded by Hive
Ranger plugin ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Hive server identity assertion
Posted by Loïc Chanel <lo...@telecomnancy.net>.
That is basically what I thought, but I didn't successfully enabled the
debug logs.
Seems like my problem was that groups are case-sensitive in Ranger plugin
for Hive, and sysadmin is not recognized as SysAdmin.
Thanks a lot for your help :-)
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-30 19:48 GMT+02:00 Alok Lal <al...@hortonworks.com>:
> Loic,
> Answer would lie in log entries before the lines that report failure and
> show exception that you have produced below. Can you turn on debugging and
> post the log? Start by turning it on for com.xasecure.authorization.hive
> level. If cause does not jump out at you then turn it up to com.xasecure.
>
>
> From: Loïc Chanel
> Reply-To: "user@ranger.incubator.apache.org"
> Date: Thursday, July 30, 2015 at 3:01 AM
> To: "user@ranger.incubator.apache.org"
>
> Subject: Re: Hive server identity assertion
>
> Sorry for my late answer, I had to work on a different problem.
> In the meantime, I realized that I am using anger 0.4, and not Ranger 0.5.
> So this problem may have been solved in Ranger 0.5.
> Here is all the the logs I get when my user toto tries to access chaneldb
> on which he should have permission to read because he belongs to the group
> sysadmin which has all the rights (including admin) on the database :
>
> 2015-07-30 11:50:49,891 INFO [HiveServer2-Handler-Pool: Thread-48]:
> parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use
> chaneldb
> 2015-07-30 11:50:50,295 INFO [HiveServer2-Handler-Pool: Thread-48]:
> parse.ParseDriver (ParseDriver.java:parse(206)) - Parse Completed
> 2015-07-30 11:50:50,297 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=parse
> start=1438249849885 end=1438249850297 duration=412
> from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,302 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
> method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(714)) - 2:
> get_database: chaneldb
> 2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(340)) - ugi=toto
> ip=unknown-ip-addr cmd=get_database: chaneldb
> 2015-07-30 11:50:50,348 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(557)) - 2: Opening
> raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2015-07-30 11:50:50,350 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.ObjectStore (ObjectStore.java:initialize(262)) - ObjectStore,
> initialize called
> 2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(131)) - Using
> direct SQL, underlying DB is MYSQL
> 2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.ObjectStore (ObjectStore.java:setConf(245)) - Initialized
> ObjectStore
> 2015-07-30 11:50:50,391 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metadata.HiveUtils
> (HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding
> metastore authorization provider:
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
> 2015-07-30 11:50:50,395 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metadata.HiveUtils
> (HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding
> metastore authorization provider:
> org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly
> 2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]:
> ql.Driver (Driver.java:compile(429)) - Semantic Analysis Completed
> 2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
> method=semanticAnalyze start=1438249850302 end=1438249850427 duration=125
> from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]:
> ql.Driver (Driver.java:getSchema(237)) - Returning Hive schema:
> Schema(fieldSchemas:null, properties:null)
> 2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
> method=doAuthorization from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,486 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
> method=doAuthorization start=1438249850440 end=1438249850486 duration=46
> from=org.apache.hadoop.hive.ql.Driver>
>
> ==> /var/log/hive/hive-server2.log <==
> FAILED: HiveAccessControlException Permission denied: user [toto] does not
> have [USE] privilege on [chaneldb]
>
> ==> /var/log/hive/hiveserver2.log <==
> 2015-07-30 11:50:50,487 ERROR [HiveServer2-Handler-Pool: Thread-48]:
> ql.Driver (SessionState.java:printError(833)) - FAILED:
> HiveAccessControlException Permission denied: user [toto] does not have
> [USE] privilege on [chaneldb]
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:
> Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
> at
> com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
> at
> org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
> at
> org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
> at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
>
> 2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=compile
> start=1438249849844 end=1438249850488 duration=644
> from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
> method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
> method=releaseLocks start=1438249850488 end=1438249850488 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2015-07-30 11:50:50,490 WARN [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(407)) -
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: HiveAccessControlException Permission denied: user
> [toto] does not have [USE] privilege on [chaneldb]
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:111)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
> at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by:
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:
> Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
> at
> com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
> at
> org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
> at
> org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
> ... 28 more
>
> And when I'm trying the command "groups" or even "hdfs groups" on the host
> running HiveServer I get "toto : nobody UsrSysAdmin SysAdmin ..."
>
> Do you or anyone else see where the problem might come from ?
> Thanks in advance,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-24 17:30 GMT+02:00 Alok Lal <al...@hortonworks.com>:
>
>> Perhaps. It is hard to say definitively without taking a look at the logs.
>>
>> From: Loïc Chanel <lo...@telecomnancy.net>
>> Reply-To: "user@ranger.incubator.apache.org" <
>> user@ranger.incubator.apache.org>
>> Date: Friday, July 24, 2015 at 8:10 AM
>> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>>
>>
>> Subject: Re: Hive server identity assertion
>>
>> Exactly !
>>
>> And I've checked the logs once again, but I can't see any groups
>> mentioned. Does this reveal a special issue ?
>>
>> Thanks,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-24 16:23 GMT+02:00 Alok Lal <al...@hortonworks.com>:
>>
>>> If the user groups couldn't be asserted, would I see a log indicating
>>> that the user cannot be impersonated (like Knox prompts) ?
>>>
>>> Yes log should show the user and group info being sent to policy
>>> engine. For authorizing. I presume you are using ranger 0.5 to connect
>>> via beeline to a hiveserver2 instance. Right? (Not that these matter,
>>> just to set context.)
>>>
>>> Thanks
>>>
>>> From: Loïc Chanel <lo...@telecomnancy.net>
>>> Reply-To: "user@ranger.incubator.apache.org" <
>>> user@ranger.incubator.apache.org>
>>> Date: Friday, July 24, 2015 at 12:53 AM
>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org
>>> >
>>> Subject: Re: Hive server identity assertion
>>>
>>> Well, that's what I thought, but the command hdfs groups returns me a
>>> group that I use for a policy giving access to a database, and as I get the
>>> message "HiveAccessControlException Permission denied" when accessing this
>>> database, I think Hive cannot assert the groups the user belongs to.
>>>
>>> I'm using Hive 0.14.0.2.2.
>>> As the problem might come from this, I think it's important to mention
>>> that the users are synchronized from a LDAP via SSSD.
>>>
>>> If the user groups couldn't be asserted, would I see a log indicating
>>> that the user cannot be impersonated (like Knox prompts) ?
>>>
>>> Thanks,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
>>>
>>>> Hive uses the same core-site.xml settings as HDFS. So if the group
>>>> mapping work in HDFS, then it should work in Hive also.
>>>>
>>>> And if the user and groups are in linux/unix, then it should have been
>>>> support out of the box.
>>>>
>>>> What version of Hive are you using? (It shouldn’t matter)
>>>>
>>>> Thanks
>>>>
>>>> Bosco
>>>>
>>>>
>>>> From: Loïc Chanel <lo...@telecomnancy.net>
>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>> user@ranger.incubator.apache.org>
>>>> Date: Thursday, July 23, 2015 at 3:10 AM
>>>> To: "user@ranger.incubator.apache.org" <
>>>> user@ranger.incubator.apache.org>
>>>> Subject: Hive server identity assertion
>>>>
>>>> Hi all,
>>>>
>>>> As I am now exploring how Ranger works with Hive, I made some policies,
>>>> but it seems that group policies are not enforced.
>>>> Therefore, I was wondering how the Ranger plugin running on Hive was
>>>> asserting the user's identity.
>>>>
>>>> I am even more surprised by the fact that I do not have any problem
>>>> with Ranger plugin working on HDFS, which is running on the exact same node.
>>>>
>>>> In parallel, I know that Know plugin, for example, runs in a totally
>>>> different way, but as it seems that, as does HBase, Hive does not provide
>>>> with any user mapping function, I thought the identity would be asserted on
>>>> the node Hive Server is running on, as if the user was a Unix one.
>>>>
>>>> Do someone as an idea about how the user groups can be founded by Hive
>>>> Ranger plugin ?
>>>> Thanks in advance,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>>
>>>
>>
>
Re: Hive server identity assertion
Posted by Alok Lal <al...@hortonworks.com>.
Loic,
Answer would lie in log entries before the lines that report failure and show exception that you have produced below. Can you turn on debugging and post the log? Start by turning it on for com.xasecure.authorization.hive level. If cause does not jump out at you then turn it up to com.xasecure.
From: Loïc Chanel
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>"
Date: Thursday, July 30, 2015 at 3:01 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>"
Subject: Re: Hive server identity assertion
Sorry for my late answer, I had to work on a different problem.
In the meantime, I realized that I am using anger 0.4, and not Ranger 0.5. So this problem may have been solved in Ranger 0.5.
Here is all the the logs I get when my user toto tries to access chaneldb on which he should have permission to read because he belongs to the group sysadmin which has all the rights (including admin) on the database :
2015-07-30 11:50:49,891 INFO [HiveServer2-Handler-Pool: Thread-48]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use chaneldb
2015-07-30 11:50:50,295 INFO [HiveServer2-Handler-Pool: Thread-48]: parse.ParseDriver (ParseDriver.java:parse(206)) - Parse Completed
2015-07-30 11:50:50,297 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=parse start=1438249849885 end=1438249850297 duration=412 from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,302 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(714)) - 2: get_database: chaneldb
2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(340)) - ugi=toto ip=unknown-ip-addr cmd=get_database: chaneldb
2015-07-30 11:50:50,348 INFO [HiveServer2-Handler-Pool: Thread-48]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(557)) - 2: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-07-30 11:50:50,350 INFO [HiveServer2-Handler-Pool: Thread-48]: metastore.ObjectStore (ObjectStore.java:initialize(262)) - ObjectStore, initialize called
2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(131)) - Using direct SQL, underlying DB is MYSQL
2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]: metastore.ObjectStore (ObjectStore.java:setConf(245)) - Initialized ObjectStore
2015-07-30 11:50:50,391 INFO [HiveServer2-Handler-Pool: Thread-48]: metadata.HiveUtils (HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding metastore authorization provider: org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
2015-07-30 11:50:50,395 INFO [HiveServer2-Handler-Pool: Thread-48]: metadata.HiveUtils (HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding metastore authorization provider: org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly
2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]: ql.Driver (Driver.java:compile(429)) - Semantic Analysis Completed
2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=semanticAnalyze start=1438249850302 end=1438249850427 duration=125 from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]: ql.Driver (Driver.java:getSchema(237)) - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=doAuthorization from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,486 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=doAuthorization start=1438249850440 end=1438249850486 duration=46 from=org.apache.hadoop.hive.ql.Driver>
==> /var/log/hive/hive-server2.log <==
FAILED: HiveAccessControlException Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
==> /var/log/hive/hiveserver2.log <==
2015-07-30 11:50:50,487 ERROR [HiveServer2-Handler-Pool: Thread-48]: ql.Driver (SessionState.java:printError(833)) - FAILED: HiveAccessControlException Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
at com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=compile start=1438249849844 end=1438249850488 duration=644 from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]: log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=releaseLocks start=1438249850488 end=1438249850488 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,490 WARN [HiveServer2-Handler-Pool: Thread-48]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(407)) - Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:111)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
at com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
... 28 more
And when I'm trying the command "groups" or even "hdfs groups" on the host running HiveServer I get "toto : nobody UsrSysAdmin SysAdmin ..."
Do you or anyone else see where the problem might come from ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-24 17:30 GMT+02:00 Alok Lal <al...@hortonworks.com>>:
Perhaps. It is hard to say definitively without taking a look at the logs.
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Friday, July 24, 2015 at 8:10 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: Hive server identity assertion
Exactly !
And I've checked the logs once again, but I can't see any groups mentioned. Does this reveal a special issue ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-24 16:23 GMT+02:00 Alok Lal <al...@hortonworks.com>>:
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Yes log should show the user and group info being sent to policy engine. For authorizing. I presume you are using ranger 0.5 to connect via beeline to a hiveserver2 instance. Right? (Not that these matter, just to set context.)
Thanks
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Friday, July 24, 2015 at 12:53 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: Hive server identity assertion
Well, that's what I thought, but the command hdfs groups returns me a group that I use for a policy giving access to a database, and as I get the message "HiveAccessControlException Permission denied" when accessing this database, I think Hive cannot assert the groups the user belongs to.
I'm using Hive 0.14.0.2.2.
As the problem might come from this, I think it's important to mention that the users are synchronized from a LDAP via SSSD.
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>>:
Hive uses the same core-site.xml settings as HDFS. So if the group mapping work in HDFS, then it should work in Hive also.
And if the user and groups are in linux/unix, then it should have been support out of the box.
What version of Hive are you using? (It shouldn’t matter)
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Thursday, July 23, 2015 at 3:10 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Hive server identity assertion
Hi all,
As I am now exploring how Ranger works with Hive, I made some policies, but it seems that group policies are not enforced.
Therefore, I was wondering how the Ranger plugin running on Hive was asserting the user's identity.
I am even more surprised by the fact that I do not have any problem with Ranger plugin working on HDFS, which is running on the exact same node.
In parallel, I know that Know plugin, for example, runs in a totally different way, but as it seems that, as does HBase, Hive does not provide with any user mapping function, I thought the identity would be asserted on the node Hive Server is running on, as if the user was a Unix one.
Do someone as an idea about how the user groups can be founded by Hive Ranger plugin ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Hive server identity assertion
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Sorry for my late answer, I had to work on a different problem.
In the meantime, I realized that I am using anger 0.4, and not Ranger 0.5.
So this problem may have been solved in Ranger 0.5.
Here is all the the logs I get when my user toto tries to access chaneldb
on which he should have permission to read because he belongs to the group
sysadmin which has all the rights (including admin) on the database :
2015-07-30 11:50:49,891 INFO [HiveServer2-Handler-Pool: Thread-48]:
parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use
chaneldb
2015-07-30 11:50:50,295 INFO [HiveServer2-Handler-Pool: Thread-48]:
parse.ParseDriver (ParseDriver.java:parse(206)) - Parse Completed
2015-07-30 11:50:50,297 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=parse
start=1438249849885 end=1438249850297 duration=412
from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,302 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]:
metastore.HiveMetaStore (HiveMetaStore.java:logInfo(714)) - 2:
get_database: chaneldb
2015-07-30 11:50:50,347 INFO [HiveServer2-Handler-Pool: Thread-48]:
HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(340)) - ugi=toto
ip=unknown-ip-addr cmd=get_database: chaneldb
2015-07-30 11:50:50,348 INFO [HiveServer2-Handler-Pool: Thread-48]:
metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(557)) - 2: Opening
raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
2015-07-30 11:50:50,350 INFO [HiveServer2-Handler-Pool: Thread-48]:
metastore.ObjectStore (ObjectStore.java:initialize(262)) - ObjectStore,
initialize called
2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]:
metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(131)) - Using
direct SQL, underlying DB is MYSQL
2015-07-30 11:50:50,371 INFO [HiveServer2-Handler-Pool: Thread-48]:
metastore.ObjectStore (ObjectStore.java:setConf(245)) - Initialized
ObjectStore
2015-07-30 11:50:50,391 INFO [HiveServer2-Handler-Pool: Thread-48]:
metadata.HiveUtils
(HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding
metastore authorization provider:
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
2015-07-30 11:50:50,395 INFO [HiveServer2-Handler-Pool: Thread-48]:
metadata.HiveUtils
(HiveUtils.java:getMetaStoreAuthorizeProviderManagers(353)) - Adding
metastore authorization provider:
org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly
2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]:
ql.Driver (Driver.java:compile(429)) - Semantic Analysis Completed
2015-07-30 11:50:50,427 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
method=semanticAnalyze start=1438249850302 end=1438249850427 duration=125
from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]:
ql.Driver (Driver.java:getSchema(237)) - Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
2015-07-30 11:50:50,440 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
method=doAuthorization from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,486 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
method=doAuthorization start=1438249850440 end=1438249850486 duration=46
from=org.apache.hadoop.hive.ql.Driver>
==> /var/log/hive/hive-server2.log <==
FAILED: HiveAccessControlException Permission denied: user [toto] does not
have [USE] privilege on [chaneldb]
==> /var/log/hive/hiveserver2.log <==
2015-07-30 11:50:50,487 ERROR [HiveServer2-Handler-Pool: Thread-48]:
ql.Driver (SessionState.java:printError(833)) - FAILED:
HiveAccessControlException Permission denied: user [toto] does not have
[USE] privilege on [chaneldb]
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:
Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
at
com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
at
org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=compile
start=1438249849844 end=1438249850488 duration=644
from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,488 INFO [HiveServer2-Handler-Pool: Thread-48]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG
method=releaseLocks start=1438249850488 end=1438249850488 duration=0
from=org.apache.hadoop.hive.ql.Driver>
2015-07-30 11:50:50,490 WARN [HiveServer2-Handler-Pool: Thread-48]:
thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(407)) -
Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: HiveAccessControlException Permission denied: user
[toto] does not have [USE] privilege on [chaneldb]
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:111)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:180)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
at com.sun.proxy.$Proxy28.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:270)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by:
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:
Permission denied: user [toto] does not have [USE] privilege on [chaneldb]
at
com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizer.checkPrivileges(XaSecureHiveAuthorizer.java:254)
at
org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:727)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:520)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1069)
at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1063)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:109)
... 28 more
And when I'm trying the command "groups" or even "hdfs groups" on the host
running HiveServer I get "toto : nobody UsrSysAdmin SysAdmin ..."
Do you or anyone else see where the problem might come from ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-24 17:30 GMT+02:00 Alok Lal <al...@hortonworks.com>:
> Perhaps. It is hard to say definitively without taking a look at the logs.
>
> From: Loïc Chanel <lo...@telecomnancy.net>
> Reply-To: "user@ranger.incubator.apache.org" <
> user@ranger.incubator.apache.org>
> Date: Friday, July 24, 2015 at 8:10 AM
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>
> Subject: Re: Hive server identity assertion
>
> Exactly !
>
> And I've checked the logs once again, but I can't see any groups
> mentioned. Does this reveal a special issue ?
>
> Thanks,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-24 16:23 GMT+02:00 Alok Lal <al...@hortonworks.com>:
>
>> If the user groups couldn't be asserted, would I see a log indicating
>> that the user cannot be impersonated (like Knox prompts) ?
>>
>> Yes log should show the user and group info being sent to policy engine.
>> For authorizing. I presume you are using ranger 0.5 to connect via beeline
>> to a hiveserver2 instance. Right? (Not that these matter, just to set
>> context.)
>>
>> Thanks
>>
>> From: Loïc Chanel <lo...@telecomnancy.net>
>> Reply-To: "user@ranger.incubator.apache.org" <
>> user@ranger.incubator.apache.org>
>> Date: Friday, July 24, 2015 at 12:53 AM
>> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>> Subject: Re: Hive server identity assertion
>>
>> Well, that's what I thought, but the command hdfs groups returns me a
>> group that I use for a policy giving access to a database, and as I get the
>> message "HiveAccessControlException Permission denied" when accessing this
>> database, I think Hive cannot assert the groups the user belongs to.
>>
>> I'm using Hive 0.14.0.2.2.
>> As the problem might come from this, I think it's important to mention
>> that the users are synchronized from a LDAP via SSSD.
>>
>> If the user groups couldn't be asserted, would I see a log indicating
>> that the user cannot be impersonated (like Knox prompts) ?
>>
>> Thanks,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
>>
>>> Hive uses the same core-site.xml settings as HDFS. So if the group
>>> mapping work in HDFS, then it should work in Hive also.
>>>
>>> And if the user and groups are in linux/unix, then it should have been
>>> support out of the box.
>>>
>>> What version of Hive are you using? (It shouldn’t matter)
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
>>> From: Loïc Chanel <lo...@telecomnancy.net>
>>> Reply-To: "user@ranger.incubator.apache.org" <
>>> user@ranger.incubator.apache.org>
>>> Date: Thursday, July 23, 2015 at 3:10 AM
>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org
>>> >
>>> Subject: Hive server identity assertion
>>>
>>> Hi all,
>>>
>>> As I am now exploring how Ranger works with Hive, I made some policies,
>>> but it seems that group policies are not enforced.
>>> Therefore, I was wondering how the Ranger plugin running on Hive was
>>> asserting the user's identity.
>>>
>>> I am even more surprised by the fact that I do not have any problem with
>>> Ranger plugin working on HDFS, which is running on the exact same node.
>>>
>>> In parallel, I know that Know plugin, for example, runs in a totally
>>> different way, but as it seems that, as does HBase, Hive does not provide
>>> with any user mapping function, I thought the identity would be asserted on
>>> the node Hive Server is running on, as if the user was a Unix one.
>>>
>>> Do someone as an idea about how the user groups can be founded by Hive
>>> Ranger plugin ?
>>> Thanks in advance,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>>
>>
>
Re: Hive server identity assertion
Posted by Alok Lal <al...@hortonworks.com>.
Perhaps. It is hard to say definitively without taking a look at the logs.
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Friday, July 24, 2015 at 8:10 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: Hive server identity assertion
Exactly !
And I've checked the logs once again, but I can't see any groups mentioned. Does this reveal a special issue ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-24 16:23 GMT+02:00 Alok Lal <al...@hortonworks.com>>:
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Yes log should show the user and group info being sent to policy engine. For authorizing. I presume you are using ranger 0.5 to connect via beeline to a hiveserver2 instance. Right? (Not that these matter, just to set context.)
Thanks
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Friday, July 24, 2015 at 12:53 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: Hive server identity assertion
Well, that's what I thought, but the command hdfs groups returns me a group that I use for a policy giving access to a database, and as I get the message "HiveAccessControlException Permission denied" when accessing this database, I think Hive cannot assert the groups the user belongs to.
I'm using Hive 0.14.0.2.2.
As the problem might come from this, I think it's important to mention that the users are synchronized from a LDAP via SSSD.
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>>:
Hive uses the same core-site.xml settings as HDFS. So if the group mapping work in HDFS, then it should work in Hive also.
And if the user and groups are in linux/unix, then it should have been support out of the box.
What version of Hive are you using? (It shouldn't matter)
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Thursday, July 23, 2015 at 3:10 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Hive server identity assertion
Hi all,
As I am now exploring how Ranger works with Hive, I made some policies, but it seems that group policies are not enforced.
Therefore, I was wondering how the Ranger plugin running on Hive was asserting the user's identity.
I am even more surprised by the fact that I do not have any problem with Ranger plugin working on HDFS, which is running on the exact same node.
In parallel, I know that Know plugin, for example, runs in a totally different way, but as it seems that, as does HBase, Hive does not provide with any user mapping function, I thought the identity would be asserted on the node Hive Server is running on, as if the user was a Unix one.
Do someone as an idea about how the user groups can be founded by Hive Ranger plugin ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Hive server identity assertion
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Exactly !
And I've checked the logs once again, but I can't see any groups mentioned.
Does this reveal a special issue ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-24 16:23 GMT+02:00 Alok Lal <al...@hortonworks.com>:
> If the user groups couldn't be asserted, would I see a log indicating
> that the user cannot be impersonated (like Knox prompts) ?
>
> Yes log should show the user and group info being sent to policy
> engine. For authorizing. I presume you are using ranger 0.5 to connect
> via beeline to a hiveserver2 instance. Right? (Not that these matter,
> just to set context.)
>
> Thanks
>
> From: Loïc Chanel <lo...@telecomnancy.net>
> Reply-To: "user@ranger.incubator.apache.org" <
> user@ranger.incubator.apache.org>
> Date: Friday, July 24, 2015 at 12:53 AM
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Re: Hive server identity assertion
>
> Well, that's what I thought, but the command hdfs groups returns me a
> group that I use for a policy giving access to a database, and as I get the
> message "HiveAccessControlException Permission denied" when accessing this
> database, I think Hive cannot assert the groups the user belongs to.
>
> I'm using Hive 0.14.0.2.2.
> As the problem might come from this, I think it's important to mention
> that the users are synchronized from a LDAP via SSSD.
>
> If the user groups couldn't be asserted, would I see a log indicating
> that the user cannot be impersonated (like Knox prompts) ?
>
> Thanks,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
>
>> Hive uses the same core-site.xml settings as HDFS. So if the group
>> mapping work in HDFS, then it should work in Hive also.
>>
>> And if the user and groups are in linux/unix, then it should have been
>> support out of the box.
>>
>> What version of Hive are you using? (It shouldn’t matter)
>>
>> Thanks
>>
>> Bosco
>>
>>
>> From: Loïc Chanel <lo...@telecomnancy.net>
>> Reply-To: "user@ranger.incubator.apache.org" <
>> user@ranger.incubator.apache.org>
>> Date: Thursday, July 23, 2015 at 3:10 AM
>> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>> Subject: Hive server identity assertion
>>
>> Hi all,
>>
>> As I am now exploring how Ranger works with Hive, I made some policies,
>> but it seems that group policies are not enforced.
>> Therefore, I was wondering how the Ranger plugin running on Hive was
>> asserting the user's identity.
>>
>> I am even more surprised by the fact that I do not have any problem with
>> Ranger plugin working on HDFS, which is running on the exact same node.
>>
>> In parallel, I know that Know plugin, for example, runs in a totally
>> different way, but as it seems that, as does HBase, Hive does not provide
>> with any user mapping function, I thought the identity would be asserted on
>> the node Hive Server is running on, as if the user was a Unix one.
>>
>> Do someone as an idea about how the user groups can be founded by Hive
>> Ranger plugin ?
>> Thanks in advance,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>>
>
Re: Hive server identity assertion
Posted by Alok Lal <al...@hortonworks.com>.
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Yes log should show the user and group info being sent to policy engine. For authorizing. I presume you are using ranger 0.5 to connect via beeline to a hiveserver2 instance. Right? (Not that these matter, just to set context.)
Thanks
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Friday, July 24, 2015 at 12:53 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: Hive server identity assertion
Well, that's what I thought, but the command hdfs groups returns me a group that I use for a policy giving access to a database, and as I get the message "HiveAccessControlException Permission denied" when accessing this database, I think Hive cannot assert the groups the user belongs to.
I'm using Hive 0.14.0.2.2.
As the problem might come from this, I think it's important to mention that the users are synchronized from a LDAP via SSSD.
If the user groups couldn't be asserted, would I see a log indicating that the user cannot be impersonated (like Knox prompts) ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>>:
Hive uses the same core-site.xml settings as HDFS. So if the group mapping work in HDFS, then it should work in Hive also.
And if the user and groups are in linux/unix, then it should have been support out of the box.
What version of Hive are you using? (It shouldn't matter)
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Thursday, July 23, 2015 at 3:10 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Hive server identity assertion
Hi all,
As I am now exploring how Ranger works with Hive, I made some policies, but it seems that group policies are not enforced.
Therefore, I was wondering how the Ranger plugin running on Hive was asserting the user's identity.
I am even more surprised by the fact that I do not have any problem with Ranger plugin working on HDFS, which is running on the exact same node.
In parallel, I know that Know plugin, for example, runs in a totally different way, but as it seems that, as does HBase, Hive does not provide with any user mapping function, I thought the identity would be asserted on the node Hive Server is running on, as if the user was a Unix one.
Do someone as an idea about how the user groups can be founded by Hive Ranger plugin ?
Thanks in advance,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Hive server identity assertion
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Well, that's what I thought, but the command hdfs groups returns me a group
that I use for a policy giving access to a database, and as I get the
message "HiveAccessControlException Permission denied" when accessing this
database, I think Hive cannot assert the groups the user belongs to.
I'm using Hive 0.14.0.2.2.
As the problem might come from this, I think it's important to mention that
the users are synchronized from a LDAP via SSSD.
If the user groups couldn't be asserted, would I see a log indicating that
the user cannot be impersonated (like Knox prompts) ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-23 20:09 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
> Hive uses the same core-site.xml settings as HDFS. So if the group mapping
> work in HDFS, then it should work in Hive also.
>
> And if the user and groups are in linux/unix, then it should have been
> support out of the box.
>
> What version of Hive are you using? (It shouldn’t matter)
>
> Thanks
>
> Bosco
>
>
> From: Loïc Chanel <lo...@telecomnancy.net>
> Reply-To: "user@ranger.incubator.apache.org" <
> user@ranger.incubator.apache.org>
> Date: Thursday, July 23, 2015 at 3:10 AM
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Hive server identity assertion
>
> Hi all,
>
> As I am now exploring how Ranger works with Hive, I made some policies,
> but it seems that group policies are not enforced.
> Therefore, I was wondering how the Ranger plugin running on Hive was
> asserting the user's identity.
>
> I am even more surprised by the fact that I do not have any problem with
> Ranger plugin working on HDFS, which is running on the exact same node.
>
> In parallel, I know that Know plugin, for example, runs in a totally
> different way, but as it seems that, as does HBase, Hive does not provide
> with any user mapping function, I thought the identity would be asserted on
> the node Hive Server is running on, as if the user was a Unix one.
>
> Do someone as an idea about how the user groups can be founded by Hive
> Ranger plugin ?
> Thanks in advance,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
>
Re: Hive server identity assertion
Posted by Don Bosco Durai <bo...@apache.org>.
Hive uses the same core-site.xml settings as HDFS. So if the group mapping
work in HDFS, then it should work in Hive also.
And if the user and groups are in linux/unix, then it should have been
support out of the box.
What version of Hive are you using? (It shouldn¹t matter)
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: "user@ranger.incubator.apache.org"
<us...@ranger.incubator.apache.org>
Date: Thursday, July 23, 2015 at 3:10 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Hive server identity assertion
> Hi all,
>
> As I am now exploring how Ranger works with Hive, I made some policies, but it
> seems that group policies are not enforced.
> Therefore, I was wondering how the Ranger plugin running on Hive was asserting
> the user's identity.
>
> I am even more surprised by the fact that I do not have any problem with
> Ranger plugin working on HDFS, which is running on the exact same node.
>
> In parallel, I know that Know plugin, for example, runs in a totally different
> way, but as it seems that, as does HBase, Hive does not provide with any user
> mapping function, I thought the identity would be asserted on the node Hive
> Server is running on, as if the user was a Unix one.
>
> Do someone as an idea about how the user groups can be founded by Hive Ranger
> plugin ?
> Thanks in advance,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne