You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Philip Zeyliger <ph...@cloudera.com> on 2018/03/27 21:50:12 UTC
Updating your minicluster to Hadoop 3.0...
Hi folks,
I just sent off https://gerrit.cloudera.org/#/c/9743/ and
https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which changes
the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry 2.0,
and so on. This change *will not* be back-ported to 2.x.
When you pull that change in, you'll need to re-build your minicluster
with, e.g., ./buildall.sh -testdata -format -notests. This will pull in the
new dependencies, format your cluster, and load up all the data. As you
know, it takes 1-2 hours.
If you want to hold off, you can elso set export
IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
Note that this choice between versions happens at build time, and CMake
depends on it. So, switching back and forth requires re-running CMake.
Please let me know if you run into any trouble. This is a big enough that
there may be some bumps on the road.
-- Philip
Re:Re:Re: Re: Updating your minicluster to Hadoop 3.0...
Posted by Quanlong Huang <hu...@126.com>.
It's hard for me to change username due to some company-specific security issues. I finally found a workaround for my problem.
The source of the impersonation invocation is in ThriftCLIService#getSessionHandle
if (cliService.getHiveConf().getBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS) &&
(userName != null)) {
...
sessionHandle = cliService.openSessionWithImpersonation(...);
} else {
sessionHandle = cliService.openSession(...);
}
So setting hive.server2.enable.doAs to false in hive-site.xml can avoid this invocatino. I modified my hive-site.xml templates in fe/src/test/resources for this and HiveServer2 launched successfully.
Just filed a JIRA for this workaround in case someone has an odd username like me: https://issues.apache.org/jira/browse/IMPALA-6789
Thanks,
Quanlong
在 2018-04-01 22:51:53，"Quanlong Huang" <hu...@126.com> 写道：
>Thanks, Philip!
>Since it's still unfixed, let me try to use another username.
>
>
>在 2018-04-01 12:35:42，"Philip Zeyliger" <ph...@cloudera.com> 写道：
>
>Hi Quanlong,
>
>
>You're running into https://issues.apache.org/jira/browse/HADOOP-7050. I found that JIRA via https://kb.informatica.com/solution/23/Pages/61/510035.aspx (and Google).
>
>
>What's surprising to me is that this code has been around in Hadoop for a long time. I think HiveServer2 must have changed in that it now invokes the impersonation code, whereas it must have not before.
>
>
>Here are some pointers:
>
>
>
>testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl: <name>hadoop.proxyuser.${USER}.hosts</name>
>
>
>
>https://github.com/apache/hadoop/blob/dc8e3432013153ac11d31d6b462aa96f8ca2c443/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/DefaultImpersonationProvider.java#L78
>
>
>
> String usersGroupsRegEx = prefixRegEx + "[^.]*(" +
> Pattern.quote(CONF_USERS) + "|" + Pattern.quote(CONF_GROUPS) + ")";
>
>
>To my eye, it looks like that regular expression is wrong and is what's disallowing usernames with periods/dots in them. (Once it's loosened, any code that's parsing these may also need to be fixed.)
>
>
>
>
>-- Philip
>
>
>
>
>
>On Fri, Mar 30, 2018 at 7:34 PM, Quanlong Huang <hu...@126.com> wrote:
>
>I failed to start the minicluster too but encountered another errors. HiveServer2 failed to launch and kept warning:
>
>
>2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Error opening session:
>java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
> at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
>Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not allowed to impersonate foo
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>
>
>I just rebase the codes and run ./buildall.sh -format
>Any thoughts?
>
>
>Thanks,
>Quanlong
>
>At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <dt...@cloudera.com> wrote:
>>I enabled full logging in my postgres that hosts the sentry and metastore
>>db and I don't see the table being created. If anyone has gone through the
>>process, can you: a) verify that relation SENTRY_ROLE exists in your
>>sentry_policy db, and b) tell me how many relations are in your policy_db.
>>
>>Thanks
>>Dimitris
>>
>>On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
>>dtsirogiannis@cloudera.com> wrote:
>>
>>> Good point. I used -format that in theory handles both the metastore and
>>> the sentry policy dB. The sentry_policy db is created and has some tables
>>> but not the SENTRY_ROLE.
>>>
>>> Dimitris
>>>
>>> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
>>>
>>>> I think I might have once fixed that using
>>>>
>>>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>>>>
>>>>
>>>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
>>>> dtsirogiannis@cloudera.com> wrote:
>>>>
>>>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
>>>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
>>>> that
>>>> > ring any bells?
>>>> >
>>>> > Thanks
>>>> > Dimitris
>>>> >
>>>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
>>>> > wrote:
>>>> >
>>>> > > Hi folks,
>>>> > >
>>>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
>>>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
>>>> changes
>>>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
>>>> 2.0,
>>>> > > and so on. This change *will not* be back-ported to 2.x.
>>>> > >
>>>> > > When you pull that change in, you'll need to re-build your minicluster
>>>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
>>>> in
>>>> > the
>>>> > > new dependencies, format your cluster, and load up all the data. As
>>>> you
>>>> > > know, it takes 1-2 hours.
>>>> > >
>>>> > > If you want to hold off, you can elso set export
>>>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>>>> > >
>>>> > > Note that this choice between versions happens at build time, and
>>>> CMake
>>>> > > depends on it. So, switching back and forth requires re-running CMake.
>>>> > >
>>>> > > Please let me know if you run into any trouble. This is a big enough
>>>> that
>>>> > > there may be some bumps on the road.
>>>> > >
>>>> > > -- Philip
>>>> > >
>>>> >
>>>>
>>>
>
>
Re:Re: Re: Updating your minicluster to Hadoop 3.0...
Posted by Quanlong Huang <hu...@126.com>.
Thanks, Philip!
Since it's still unfixed, let me try to use another username.
在 2018-04-01 12:35:42，"Philip Zeyliger" <ph...@cloudera.com> 写道：
Hi Quanlong,
You're running into https://issues.apache.org/jira/browse/HADOOP-7050. I found that JIRA via https://kb.informatica.com/solution/23/Pages/61/510035.aspx (and Google).
What's surprising to me is that this code has been around in Hadoop for a long time. I think HiveServer2 must have changed in that it now invokes the impersonation code, whereas it must have not before.
Here are some pointers:
testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl: <name>hadoop.proxyuser.${USER}.hosts</name>
https://github.com/apache/hadoop/blob/dc8e3432013153ac11d31d6b462aa96f8ca2c443/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/DefaultImpersonationProvider.java#L78
String usersGroupsRegEx = prefixRegEx + "[^.]*(" +
Pattern.quote(CONF_USERS) + "|" + Pattern.quote(CONF_GROUPS) + ")";
To my eye, it looks like that regular expression is wrong and is what's disallowing usernames with periods/dots in them. (Once it's loosened, any code that's parsing these may also need to be fixed.)
-- Philip
On Fri, Mar 30, 2018 at 7:34 PM, Quanlong Huang <hu...@126.com> wrote:
I failed to start the minicluster too but encountered another errors. HiveServer2 failed to launch and kept warning:
2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Error opening session:
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
... 21 more
Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not allowed to impersonate foo
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
... 21 more
I just rebase the codes and run ./buildall.sh -format
Any thoughts?
Thanks,
Quanlong
At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <dt...@cloudera.com> wrote:
>I enabled full logging in my postgres that hosts the sentry and metastore
>db and I don't see the table being created. If anyone has gone through the
>process, can you: a) verify that relation SENTRY_ROLE exists in your
>sentry_policy db, and b) tell me how many relations are in your policy_db.
>
>Thanks
>Dimitris
>
>On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
>dtsirogiannis@cloudera.com> wrote:
>
>> Good point. I used -format that in theory handles both the metastore and
>> the sentry policy dB. The sentry_policy db is created and has some tables
>> but not the SENTRY_ROLE.
>>
>> Dimitris
>>
>> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
>>
>>> I think I might have once fixed that using
>>>
>>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>>>
>>>
>>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
>>> dtsirogiannis@cloudera.com> wrote:
>>>
>>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
>>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
>>> that
>>> > ring any bells?
>>> >
>>> > Thanks
>>> > Dimitris
>>> >
>>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Hi folks,
>>> > >
>>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
>>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
>>> changes
>>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
>>> 2.0,
>>> > > and so on. This change *will not* be back-ported to 2.x.
>>> > >
>>> > > When you pull that change in, you'll need to re-build your minicluster
>>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
>>> in
>>> > the
>>> > > new dependencies, format your cluster, and load up all the data. As
>>> you
>>> > > know, it takes 1-2 hours.
>>> > >
>>> > > If you want to hold off, you can elso set export
>>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>>> > >
>>> > > Note that this choice between versions happens at build time, and
>>> CMake
>>> > > depends on it. So, switching back and forth requires re-running CMake.
>>> > >
>>> > > Please let me know if you run into any trouble. This is a big enough
>>> that
>>> > > there may be some bumps on the road.
>>> > >
>>> > > -- Philip
>>> > >
>>> >
>>>
>>
Re: Re: Updating your minicluster to Hadoop 3.0...
Posted by Philip Zeyliger <ph...@cloudera.com>.
Hi Quanlong,
You're running into https://issues.apache.org/jira/browse/HADOOP-7050. I
found that JIRA via
https://kb.informatica.com/solution/23/Pages/61/510035.aspx (and Google).
What's surprising to me is that this code has been around in Hadoop for a
long time. I think HiveServer2 must have changed in that it now invokes the
impersonation code, whereas it must have not before.
Here are some pointers:
testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl:
<name>hadoop.proxyuser.${USER}.hosts</name>
https://github.com/apache/hadoop/blob/dc8e3432013153ac11d31d6b462aa96f8ca2c443/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/DefaultImpersonationProvider.java#L78
String usersGroupsRegEx = prefixRegEx + "[^.]*(" +
Pattern.quote(CONF_USERS) + "|" + Pattern.quote(CONF_GROUPS) + ")";
To my eye, it looks like that regular expression is wrong and is what's
disallowing usernames with periods/dots in them. (Once it's loosened, any
code that's parsing these may also need to be fixed.)
-- Philip
On Fri, Mar 30, 2018 at 7:34 PM, Quanlong Huang <hu...@126.com>
wrote:
> I failed to start the minicluster too but encountered another errors.
> HiveServer2 failed to launch and kept warning:
>
> 2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49]
> thrift.ThriftCLIService: Error opening session:
> java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.security.authorize.AuthorizationException): User: quanlong.huang
> is not allowed to impersonate foo
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:89) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422)
> ~[?:1.8.0_121]
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1962) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
> at org.apache.hive.service.cli.session.SessionManager.
> createSession(SessionManager.java:411) [hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.SessionManager.
> openSession(SessionManager.java:363) [hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.CLIService.
> openSessionWithImpersonation(CLIService.java:189)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.thrift.ThriftCLIService.
> getSessionHandle(ThriftCLIService.java:423) [hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.thrift.ThriftCLIService.
> OpenSession(ThriftCLIService.java:312) [hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$
> OpenSession.getResult(TCLIService.java:1377) [hive-exec-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.rpc.thrift.TCLIService$Processor$
> OpenSession.getResult(TCLIService.java:1362) [hive-exec-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.auth.TSetIpAddressProcessor.process(
> TSetIpAddressProcessor.java:56) [hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_121]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.
> RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: quanlong.huang is not allowed to impersonate foo
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
> Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is
> not allowed to impersonate foo
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolTranslatorPB.getFileInfo(
> ClientNamenodeProtocolTranslatorPB.java:875) ~[hadoop-hdfs-client-3.0.0-
> cdh6.x-20180302.192732-2.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invokeMethod(RetryInvocationHandler.java:165)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.
> doCall(DistributedFileSystem.java:1494) ~[hadoop-hdfs-client-3.0.0-
> cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.
> doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0-
> cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0-cdh6.x-
> 20180302.191654-1.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:1506) ~[hadoop-hdfs-client-3.0.0-
> cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.hive.ql.session.SessionState.
> createRootHDFSDir(SessionState.java:701) ~[hive-exec-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.
> createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>
> I just rebase the codes and run ./buildall.sh -format
> Any thoughts?
>
> Thanks,
> Quanlong
>
>
> At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <dt...@cloudera.com> wrote:
> >I enabled full logging in my postgres that hosts the sentry and metastore
> >db and I don't see the table being created. If anyone has gone through the
> >process, can you: a) verify that relation SENTRY_ROLE exists in your
> >sentry_policy db, and b) tell me how many relations are in your policy_db.
> >
> >Thanks
> >Dimitris
> >
> >On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
> >dtsirogiannis@cloudera.com> wrote:
> >
> >> Good point. I used -format that in theory handles both the metastore and
> >> the sentry policy dB. The sentry_policy db is created and has some tables
> >> but not the SENTRY_ROLE.
> >>
> >> Dimitris
> >>
> >> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
> >>
> >>> I think I might have once fixed that using
> >>>
> >>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
> >>>
> >>>
> >>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
> >>> dtsirogiannis@cloudera.com> wrote:
> >>>
> >>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
> >>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
> >>> that
> >>> > ring any bells?
> >>> >
> >>> > Thanks
> >>> > Dimitris
> >>> >
> >>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
> >>> > wrote:
> >>> >
> >>> > > Hi folks,
> >>> > >
> >>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> >>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
> >>> changes
> >>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
> >>> 2.0,
> >>> > > and so on. This change *will not* be back-ported to 2.x.
> >>> > >
> >>> > > When you pull that change in, you'll need to re-build your minicluster
> >>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
> >>> in
> >>> > the
> >>> > > new dependencies, format your cluster, and load up all the data. As
> >>> you
> >>> > > know, it takes 1-2 hours.
> >>> > >
> >>> > > If you want to hold off, you can elso set export
> >>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> >>> > >
> >>> > > Note that this choice between versions happens at build time, and
> >>> CMake
> >>> > > depends on it. So, switching back and forth requires re-running CMake.
> >>> > >
> >>> > > Please let me know if you run into any trouble. This is a big enough
> >>> that
> >>> > > there may be some bumps on the road.
> >>> > >
> >>> > > -- Philip
> >>> > >
> >>> >
> >>>
> >>
>
>
Re:Re: Updating your minicluster to Hadoop 3.0...
Posted by Quanlong Huang <hu...@126.com>.
I failed to start the minicluster too but encountered another errors. HiveServer2 failed to launch and kept warning:
2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Error opening session:
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
... 21 more
Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not allowed to impersonate foo
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
... 21 more
I just rebase the codes and run ./buildall.sh -format
Any thoughts?
Thanks,
Quanlong
At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <dt...@cloudera.com> wrote:
>I enabled full logging in my postgres that hosts the sentry and metastore
>db and I don't see the table being created. If anyone has gone through the
>process, can you: a) verify that relation SENTRY_ROLE exists in your
>sentry_policy db, and b) tell me how many relations are in your policy_db.
>
>Thanks
>Dimitris
>
>On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
>dtsirogiannis@cloudera.com> wrote:
>
>> Good point. I used -format that in theory handles both the metastore and
>> the sentry policy dB. The sentry_policy db is created and has some tables
>> but not the SENTRY_ROLE.
>>
>> Dimitris
>>
>> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
>>
>>> I think I might have once fixed that using
>>>
>>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>>>
>>>
>>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
>>> dtsirogiannis@cloudera.com> wrote:
>>>
>>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
>>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
>>> that
>>> > ring any bells?
>>> >
>>> > Thanks
>>> > Dimitris
>>> >
>>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Hi folks,
>>> > >
>>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
>>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
>>> changes
>>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
>>> 2.0,
>>> > > and so on. This change *will not* be back-ported to 2.x.
>>> > >
>>> > > When you pull that change in, you'll need to re-build your minicluster
>>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
>>> in
>>> > the
>>> > > new dependencies, format your cluster, and load up all the data. As
>>> you
>>> > > know, it takes 1-2 hours.
>>> > >
>>> > > If you want to hold off, you can elso set export
>>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>>> > >
>>> > > Note that this choice between versions happens at build time, and
>>> CMake
>>> > > depends on it. So, switching back and forth requires re-running CMake.
>>> > >
>>> > > Please let me know if you run into any trouble. This is a big enough
>>> that
>>> > > there may be some bumps on the road.
>>> > >
>>> > > -- Philip
>>> > >
>>> >
>>>
>>
Re: Updating your minicluster to Hadoop 3.0...
Posted by Philip Zeyliger <ph...@cloudera.com>.
Hi Dimitris,
Here's a listing from my machine after I ran "./buildall.sh
-start_minicluster -start_impala_cluster -format -testdata -notests".
-- Philip
$sudo -u postgres psql
psql (9.5.12)
Type "help" for help.
postgres=# \c sentry_policy
You are now connected to database "sentry_policy" as user "postgres".
sentry_policy=# select * from "SENTRY_ROLE";
ROLE_ID | CREATE_TIME | ROLE_NAME
---------+-------------+-----------
(0 rows)
sentry_policy=# select * from "SENTRY_VERSION";
VER_ID | SCHEMA_VERSION | VERSION_COMMENT
--------+----------------+-------------------------------
1 | 2.0.0 | Schema version set implicitly
(1 row)
sentry_policy=# select * from "SENTRY_USER";
USER_ID | CREATE_TIME | USER_NAME
---------+-------------+-----------
(0 rows)
sentry_policy=# \dt
List of relations
Schema | Name | Type | Owner
--------+------------------------------+-------+----------
public | AUTHZ_PATH | table | hiveuser
public | AUTHZ_PATHS_MAPPING | table | hiveuser
public | AUTHZ_PATHS_SNAPSHOT_ID | table | hiveuser
public | SENTRY_DB_PRIVILEGE | table | hiveuser
public | SENTRY_GM_PRIVILEGE | table | hiveuser
public | SENTRY_GROUP | table | hiveuser
public | SENTRY_HMS_NOTIFICATION_ID | table | hiveuser
public | SENTRY_PATH_CHANGE | table | hiveuser
public | SENTRY_PERM_CHANGE | table | hiveuser
public | SENTRY_ROLE | table | hiveuser
public | SENTRY_ROLE_DB_PRIVILEGE_MAP | table | hiveuser
public | SENTRY_ROLE_GM_PRIVILEGE_MAP | table | hiveuser
public | SENTRY_ROLE_GROUP_MAP | table | hiveuser
public | SENTRY_ROLE_USER_MAP | table | hiveuser
public | SENTRY_USER | table | hiveuser
public | SENTRY_VERSION | table | hiveuser
public | SEQUENCE_TABLE | table | hiveuser
(17 rows)
On Thu, Mar 29, 2018 at 10:08 PM, Dimitris Tsirogiannis <
dtsirogiannis@cloudera.com> wrote:
> I enabled full logging in my postgres that hosts the sentry and metastore
> db and I don't see the table being created. If anyone has gone through the
> process, can you: a) verify that relation SENTRY_ROLE exists in your
> sentry_policy db, and b) tell me how many relations are in your policy_db.
>
> Thanks
> Dimitris
>
> On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
> dtsirogiannis@cloudera.com> wrote:
>
> > Good point. I used -format that in theory handles both the metastore and
> > the sentry policy dB. The sentry_policy db is created and has some tables
> > but not the SENTRY_ROLE.
> >
> > Dimitris
> >
> > On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
> >
> >> I think I might have once fixed that using
> >>
> >> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
> >>
> >>
> >> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
> >> dtsirogiannis@cloudera.com> wrote:
> >>
> >> > I tried rebuilding my minicluster but Sentry refuses to start. I get
> >> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
> >> that
> >> > ring any bells?
> >> >
> >> > Thanks
> >> > Dimitris
> >> >
> >> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <philip@cloudera.com
> >
> >> > wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> >> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
> >> changes
> >> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
> >> 2.0,
> >> > > and so on. This change *will not* be back-ported to 2.x.
> >> > >
> >> > > When you pull that change in, you'll need to re-build your
> minicluster
> >> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
> >> in
> >> > the
> >> > > new dependencies, format your cluster, and load up all the data. As
> >> you
> >> > > know, it takes 1-2 hours.
> >> > >
> >> > > If you want to hold off, you can elso set export
> >> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> >> > >
> >> > > Note that this choice between versions happens at build time, and
> >> CMake
> >> > > depends on it. So, switching back and forth requires re-running
> CMake.
> >> > >
> >> > > Please let me know if you run into any trouble. This is a big enough
> >> that
> >> > > there may be some bumps on the road.
> >> > >
> >> > > -- Philip
> >> > >
> >> >
> >>
> >
>
Re: Updating your minicluster to Hadoop 3.0...
Posted by Dimitris Tsirogiannis <dt...@cloudera.com>.
I enabled full logging in my postgres that hosts the sentry and metastore
db and I don't see the table being created. If anyone has gone through the
process, can you: a) verify that relation SENTRY_ROLE exists in your
sentry_policy db, and b) tell me how many relations are in your policy_db.
Thanks
Dimitris
On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
dtsirogiannis@cloudera.com> wrote:
> Good point. I used -format that in theory handles both the metastore and
> the sentry policy dB. The sentry_policy db is created and has some tables
> but not the SENTRY_ROLE.
>
> Dimitris
>
> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
>
>> I think I might have once fixed that using
>>
>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>>
>>
>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
>> dtsirogiannis@cloudera.com> wrote:
>>
>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
>> that
>> > ring any bells?
>> >
>> > Thanks
>> > Dimitris
>> >
>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
>> > wrote:
>> >
>> > > Hi folks,
>> > >
>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
>> changes
>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
>> 2.0,
>> > > and so on. This change *will not* be back-ported to 2.x.
>> > >
>> > > When you pull that change in, you'll need to re-build your minicluster
>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
>> in
>> > the
>> > > new dependencies, format your cluster, and load up all the data. As
>> you
>> > > know, it takes 1-2 hours.
>> > >
>> > > If you want to hold off, you can elso set export
>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>> > >
>> > > Note that this choice between versions happens at build time, and
>> CMake
>> > > depends on it. So, switching back and forth requires re-running CMake.
>> > >
>> > > Please let me know if you run into any trouble. This is a big enough
>> that
>> > > there may be some bumps on the road.
>> > >
>> > > -- Philip
>> > >
>> >
>>
>
Re: Updating your minicluster to Hadoop 3.0...
Posted by Dimitris Tsirogiannis <dt...@cloudera.com>.
Good point. I used -format that in theory handles both the metastore and
the sentry policy dB. The sentry_policy db is created and has some tables
but not the SENTRY_ROLE.
Dimitris
On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jb...@cloudera.com> wrote:
> I think I might have once fixed that using
>
> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>
>
> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
> dtsirogiannis@cloudera.com> wrote:
>
> > I tried rebuilding my minicluster but Sentry refuses to start. I get
> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
> that
> > ring any bells?
> >
> > Thanks
> > Dimitris
> >
> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
> changes
> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
> 2.0,
> > > and so on. This change *will not* be back-ported to 2.x.
> > >
> > > When you pull that change in, you'll need to re-build your minicluster
> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull in
> > the
> > > new dependencies, format your cluster, and load up all the data. As you
> > > know, it takes 1-2 hours.
> > >
> > > If you want to hold off, you can elso set export
> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> > >
> > > Note that this choice between versions happens at build time, and CMake
> > > depends on it. So, switching back and forth requires re-running CMake.
> > >
> > > Please let me know if you run into any trouble. This is a big enough
> that
> > > there may be some bumps on the road.
> > >
> > > -- Philip
> > >
> >
>
Re: Updating your minicluster to Hadoop 3.0...
Posted by Jim Apple <jb...@cloudera.com>.
I think I might have once fixed that using
./buildall.sh -notests -format_metastore -format_sentry_policy_db
On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
dtsirogiannis@cloudera.com> wrote:
> I tried rebuilding my minicluster but Sentry refuses to start. I get
> "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does that
> ring any bells?
>
> Thanks
> Dimitris
>
> On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > Hi folks,
> >
> > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which changes
> > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry 2.0,
> > and so on. This change *will not* be back-ported to 2.x.
> >
> > When you pull that change in, you'll need to re-build your minicluster
> > with, e.g., ./buildall.sh -testdata -format -notests. This will pull in
> the
> > new dependencies, format your cluster, and load up all the data. As you
> > know, it takes 1-2 hours.
> >
> > If you want to hold off, you can elso set export
> > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> >
> > Note that this choice between versions happens at build time, and CMake
> > depends on it. So, switching back and forth requires re-running CMake.
> >
> > Please let me know if you run into any trouble. This is a big enough that
> > there may be some bumps on the road.
> >
> > -- Philip
> >
>
Re: Updating your minicluster to Hadoop 3.0...
Posted by Dimitris Tsirogiannis <dt...@cloudera.com>.
I tried rebuilding my minicluster but Sentry refuses to start. I get
"ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does that
ring any bells?
Thanks
Dimitris
On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
wrote:
> Hi folks,
>
> I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which changes
> the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry 2.0,
> and so on. This change *will not* be back-ported to 2.x.
>
> When you pull that change in, you'll need to re-build your minicluster
> with, e.g., ./buildall.sh -testdata -format -notests. This will pull in the
> new dependencies, format your cluster, and load up all the data. As you
> know, it takes 1-2 hours.
>
> If you want to hold off, you can elso set export
> IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>
> Note that this choice between versions happens at build time, and CMake
> depends on it. So, switching back and forth requires re-running CMake.
>
> Please let me know if you run into any trouble. This is a big enough that
> there may be some bumps on the road.
>
> -- Philip
>