You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Garry Chen <gc...@cornell.edu> on 2018/11/07 19:11:53 UTC
Create external table with s3 location error
hi All,
I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion?
hive> create external table kv (key int, values string) location 's3://cu-iclick/test';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException)
Garry
Re: Create external table with s3 location error
Posted by Garry Chen <gc...@cornell.edu>.
Yes, it works. Thank you very much,
Garry
From: Suresh Kumar Sethuramaswamy <ro...@gmail.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
Date: Wednesday, November 7, 2018 at 3:10 PM
To: "user@hive.apache.org" <us...@hive.apache.org>
Subject: Re: Create external table with s3 location error
Thanks for the logs. Couple of things here,
1. Based on the logs,the HiveServer2 seem to be down. This is an issue if you use Hue/Beeline
2. If you are using Hive CLI, you can still test it, where you have to add the (fs.s3a.access.key,fs.s3.secret.key)s3 keys in hdfs advanced configuration snippet for core-site.xml and hdfs-site.xml via CM.Also for location it should be s3a:// not s3://
On Wed, Nov 7, 2018, 2:58 PM Garry Chen <gc...@cornell.edu> wrote:
Hi Suresh,
I am using Hive 1.1.0-cdh5.14.4 and hive server log as below.
2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
... 17 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
... 21 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
... 27 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
... 34 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 42 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
From: Suresh Kumar Sethuramaswamy <ro...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Wednesday, November 7, 2018 at 2:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Create external table with s3 location error
Are you using EMR or Apache hadoop open source?
Can you share your hive megastore logs?
On Wed, Nov 7, 2018, 2:19 PM Garry Chen <gc...@cornell.edu> wrote:
hi All,
I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion?
hive> create external table kv (key int, values string) location 's3://cu-iclick/test';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException)
Garry
Re: Create external table with s3 location error
Posted by Suresh Kumar Sethuramaswamy <ro...@gmail.com>.
Thanks for the logs. Couple of things here,
1. Based on the logs,the HiveServer2 seem to be down. This is an issue if
you use Hue/Beeline
2. If you are using Hive CLI, you can still test it, where you have to add
the (fs.s3a.access.key,fs.s3.secret.key)s3 keys in hdfs advanced
configuration snippet for core-site.xml and hdfs-site.xml via CM.Also for
location it should be s3a:// not s3://
On Wed, Nov 7, 2018, 2:58 PM Garry Chen <gc92@cornell.edu wrote:
> Hi Suresh,
>
> I am using Hive 1.1.0-cdh5.14.4 and hive server log as below.
>
>
>
> 2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2
> (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on
> attempt 1, will retry in 60000ms
>
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
>
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
>
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
>
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
>
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
>
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
>
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
>
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
>
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
>
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>
> at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
>
> ... 17 more
>
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>
> at
> org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
>
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>
> at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
>
> ... 18 more
>
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
>
> at
> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
>
> at
> org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
>
> ... 21 more
>
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
>
> ... 27 more
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
>
> ... 34 more
>
> Caused by: MetaException(message:Could not connect to meta store using any
> of the URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
>
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
>
> at
> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
>
> at
> org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
>
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>
> at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
>
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
>
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
>
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
>
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
>
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
>
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
>
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
>
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
>
> Caused by: java.net.ConnectException: Connection refused (Connection
> refused)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
>
> ... 42 more
>
> )
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
>
>
>
> *From: *Suresh Kumar Sethuramaswamy <ro...@gmail.com>
> *Reply-To: *"user@hive.apache.org" <us...@hive.apache.org>
> *Date: *Wednesday, November 7, 2018 at 2:50 PM
> *To: *"user@hive.apache.org" <us...@hive.apache.org>
> *Subject: *Re: Create external table with s3 location error
>
>
>
> Are you using EMR or Apache hadoop open source?
>
> Can you share your hive megastore logs?
>
>
>
>
>
> On Wed, Nov 7, 2018, 2:19 PM Garry Chen <gc92@cornell.edu wrote:
>
> hi All,
>
> I am try to create a external table using s3 as location
> but failed. I add my access key and security key in hive-site.xml and
> reboot the server. Any suggestion?
>
>
>
> hive> create external table kv (key int, values string) location
> 's3://cu-iclick/test';
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
>
>
>
> Garry
>
>
Re: Create external table with s3 location error
Posted by Garry Chen <gc...@cornell.edu>.
Hi Suresh,
I am using Hive 1.1.0-cdh5.14.4 and hive server log as below.
2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
... 17 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
... 21 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
... 27 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
... 34 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256)
at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29)
at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:111)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 42 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
From: Suresh Kumar Sethuramaswamy <ro...@gmail.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
Date: Wednesday, November 7, 2018 at 2:50 PM
To: "user@hive.apache.org" <us...@hive.apache.org>
Subject: Re: Create external table with s3 location error
Are you using EMR or Apache hadoop open source?
Can you share your hive megastore logs?
On Wed, Nov 7, 2018, 2:19 PM Garry Chen <gc...@cornell.edu> wrote:
hi All,
I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion?
hive> create external table kv (key int, values string) location 's3://cu-iclick/test';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException)
Garry
Re: Create external table with s3 location error
Posted by Suresh Kumar Sethuramaswamy <ro...@gmail.com>.
Are you using EMR or Apache hadoop open source?
Can you share your hive megastore logs?
On Wed, Nov 7, 2018, 2:19 PM Garry Chen <gc92@cornell.edu wrote:
> hi All,
>
> I am try to create a external table using s3 as location
> but failed. I add my access key and security key in hive-site.xml and
> reboot the server. Any suggestion?
>
>
>
> hive> create external table kv (key int, values string) location
> 's3://cu-iclick/test';
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
>
>
>
> Garry
>