You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Erin Boyd <eb...@redhat.com> on 2014/10/28 16:52:09 UTC
Hive metastore cannot start
Hi,
Has anyone else had an issue where the Hive Server and mysql will start but not the hive metastore?
I am getting this in the hiveserver2.log logs:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
at org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96)
at org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:109)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
... 13 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
... 18 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
This is running local (not remote). THe URI has been the FQDN, locallost, 0.0.0.0 -> yet I still get the same error.
I have tried changing ports...
Anyone seen this and knows how to fix/debug it?
Erin
Re: Hive metastore cannot start
Posted by John Speidel <js...@hortonworks.com>.
Glad that you git it resolved :)
On Tue, Oct 28, 2014 at 2:33 PM, Erin Boyd <eb...@redhat.com> wrote:
> Hi John,
> So I got it running.
> I restarted the server and agent and then restarted Hive and it suddenly
> works.
> I do believe increasing the heap size was necessary, I am just not sure it
> was picking it up for the JVM until the server reboot.
>
> Erin
>
>
> ----- Original Message -----
> From: "Erin Boyd" <eb...@redhat.com>
> To: dev@ambari.apache.org
> Sent: Tuesday, October 28, 2014 11:53:22 AM
> Subject: Re: Hive metastore cannot start
>
> HI John,
> I also tried to start it via the command line
> [root@ibm-x3650m4-02-vm-02 hive]# hive --service metastore
> Starting Hive Metastore Server
> Exception in thread "pool-3-thread-31" java.lang.OutOfMemoryError: Java
> heap space
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Exception in thread "pool-3-thread-33" java.lang.OutOfMemoryError: Java
> heap space
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Exception in thread "pool-3-thread-36" java.lang.OutOfMemoryError: Java
> heap space
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> So from this I increased the heap size to 2048 (it was 1024).
>
>
> It did however allow the listener to start:
> [root@ibm-x3650m4-02-vm-02 conf]# netstat -an | grep 9083
> tcp 0 0 0.0.0.0:9083 0.0.0.0:*
> LISTEN
> tcp 0 0 10.16.40.239:59522 10.16.40.239:9083
> TIME_WAIT
> tcp 0 0 10.16.40.239:59563 10.16.40.239:9083
> TIME_WAIT
> [root@ibm-x3650m4-02-vm-02 conf]#
>
> If I start it via the UI, I get:
> Python script has been killed due to timeout
>
> But if I look in the hiveserver2.log I see this:
> 2014-10-28 12:53:13,615 INFO [main]: hive.metastore
> (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with
> URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
> 2014-10-28 12:53:13,735 INFO [main]: hive.metastore
> (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
> 2014-10-28 12:53:13,954 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
> 2014-10-28 12:53:13,955 INFO [main]: glusterfs.GlusterFileSystem
> (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
> 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem
> (GlusterFileSystem.java:<init>(51)) - Initializing GlusterFS, CRC disabled.
> 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem
> (GlusterFileSystem.java:<init>(52)) - GIT
> INFO={git.commit.id.abbrev=acca0e1, git.commit.user.email=
> bchilds@sh-el6.eng.rdu2.redhat.com, git.commit.message.full=[update RPM
> spec file/changelog] - 2.3.3
> , git.commit.id=acca0e11de5f49a67474af70f6758d4182852760,
> git.commit.message.short=[update RPM spec file/changelog] - 2.3.3,
> git.commit.user.name=Brad Childs, git.build.user.name=Unknown,
> git.commit.id.describe=2.3.10-12-gacca0e1, git.build.user.email=Unknown,
> git.branch=2.3.3, git.commit.time=29.07.2014 @ 10:52:06 EDT,
> git.build.time=29.07.2014 @ 11:06:36 EDT}
> 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem
> (GlusterFileSystem.java:<init>(53)) - GIT_TAG=2.3.10
> 2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem
> (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
> 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
> 2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol1 at :
> /mnt/glusterfs/HadoopVol1
> 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(121)) - Gluster volume: gv0 at : /mnt/gv0
> 2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol2 at :
> /mnt/glusterfs/HadoopVol2
> 2014-10-28 12:53:14,053 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(155)) - Working directory is :
> glusterfs:/user/hive
> 2014-10-28 12:53:14,054 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(164)) - Write buffer size : 131072
> 2014-10-28 12:53:14,055 INFO [main]: glusterfs.GlusterVolume
> (GlusterVolume.java:setConf(175)) - Default block size : 67108864
> 2014-10-28 12:53:14,056 INFO [main]: session.SessionState
> (SessionState.java:start(361)) - No Tez session required at this point.
> hive.execution.engine=mr.
> 2014-10-28 12:53:14,070 INFO [main]: service.CompositeService
> (SessionManager.java:createBackgroundOperationPool(76)) - HiveServer2:
> Background operation thread pool size: 100
> 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService
> (SessionManager.java:createBackgroundOperationPool(78)) - HiveServer2:
> Background operation thread wait queue size: 100
> 2014-10-28 12:53:14,071 INFO [main]: service.CompositeService
> (SessionManager.java:createBackgroundOperationPool(80)) - HiveServer2:
> Background operation thread keepalive time: 10
> 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService
> (AbstractService.java:init(89)) - Service:OperationManager is inited.
> 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService
> (AbstractService.java:init(89)) - Service:SessionManager is inited.
> 2014-10-28 12:53:14,074 INFO [main]: service.AbstractService
> (AbstractService.java:init(89)) - Service:CLIService is inited.
> 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService
> (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
> 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService
> (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
> 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService
> (AbstractService.java:start(104)) - Service:OperationManager is started.
> 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService
> (AbstractService.java:start(104)) - Service:SessionManager is started.
> 2014-10-28 12:53:14,075 INFO [main]: service.AbstractService
> (AbstractService.java:start(104)) - Service:CLIService is started.
> 2014-10-28 12:53:14,130 INFO [main]: hive.metastore
> (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with
> URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
> 2014-10-28 12:53:14,133 INFO [main]: hive.metastore
> (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
> 2014-10-28 12:53:14,209 INFO [main]: service.AbstractService
> (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is
> started.
> 2014-10-28 12:53:14,210 INFO [main]: service.AbstractService
> (AbstractService.java:start(104)) - Service:HiveServer2 is started.
> 2014-10-28 12:53:14,399 INFO [Thread-4]: thrift.ThriftCLIService
> (ThriftBinaryCLIService.java:run(97)) - ThriftBinaryCLIService listening on
> 0.0.0.0/0.0.0.0:10000
>
>
> which makes it appear that it's working....
>
> also this:
> [root@ibm-x3650m4-02-vm-02 root]# ps -ef | grep hive
> hcat 9251 1 0 Oct27 ? 00:02:31
> /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true
> -Dwebhcat.log.dir=/var/log/webhcat/
> -Dlog4j.configuration=file:///usr/lib/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties
> -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hcat
> -Dhadoop.root.logger=INFO,console
> -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.util.RunJar
> /usr/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-0.13.0.2.1.7.0-784.jar
> org.apache.hive.hcatalog.templeton.Main
> root 9808 10146 0 13:14 pts/1 00:00:10
> /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=root
> -Dhadoop.root.logger=INFO,console
> -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.util.RunJar
> /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar
> org.apache.hadoop.hive.metastore.HiveMetaStore
> root 20972 7782 0 13:32 pts/4 00:00:00 grep hive
> hive 27185 1 0 12:49 ? 00:00:23
> /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive
> -Dhadoop.root.logger=INFO,console
> -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.util.RunJar
> /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar
> org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris="
> " -hiveconf hive.log.file=hiveserver2.log -hiveconf
> hive.log.dir=/var/log/hive
> [root@ibm-x3650m4-02-vm-02 root]#
>
>
> Hum........
>
> But if you look at the UI, it still says stopped.
>
> Erin
>
>
>
> ----- Original Message -----
> From: "John Speidel" <js...@hortonworks.com>
> To: dev@ambari.apache.org
> Sent: Tuesday, October 28, 2014 10:33:12 AM
> Subject: Re: Hive metastore cannot start
>
> Erin,
>
> Please provide an exported blueprint for your cluster so that we can take a
> look at your configs.
>
> http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint
>
> Thanks,
> John
>
> On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <eb...@redhat.com> wrote:
>
> > Hi,
> > Has anyone else had an issue where the Hive Server and mysql will start
> > but not the hive metastore?
> > I am getting this in the hiveserver2.log logs:
> >
> > java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> > instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at
> >
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
> > at
> >
> org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96)
> > at
> >
> org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64)
> > at
> > org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> > at
> org.apache.hive.service.cli.CLIService.init(CLIService.java:109)
> > at
> > org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> > at
> > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
> > at
> >
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
> > at
> > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > Caused by: java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> > at
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
> > at
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> > at
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> > at
> >
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
> > at
> >
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
> > ... 13 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
> > ... 18 more
> > Caused by: MetaException(message:Could not connect to meta store using
> any
> > of the URIs provided. Most recent failure:
> > org.apache.thrift.transport.TTransportException:
> java.net.ConnectException:
> > Connection refused
> >
> > This is running local (not remote). THe URI has been the FQDN, locallost,
> > 0.0.0.0 -> yet I still get the same error.
> > I have tried changing ports...
> >
> > Anyone seen this and knows how to fix/debug it?
> >
> > Erin
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hive metastore cannot start
Posted by Erin Boyd <eb...@redhat.com>.
Hi John,
So I got it running.
I restarted the server and agent and then restarted Hive and it suddenly works.
I do believe increasing the heap size was necessary, I am just not sure it was picking it up for the JVM until the server reboot.
Erin
----- Original Message -----
From: "Erin Boyd" <eb...@redhat.com>
To: dev@ambari.apache.org
Sent: Tuesday, October 28, 2014 11:53:22 AM
Subject: Re: Hive metastore cannot start
HI John,
I also tried to start it via the command line
[root@ibm-x3650m4-02-vm-02 hive]# hive --service metastore
Starting Hive Metastore Server
Exception in thread "pool-3-thread-31" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-3-thread-33" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-3-thread-36" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
So from this I increased the heap size to 2048 (it was 1024).
It did however allow the listener to start:
[root@ibm-x3650m4-02-vm-02 conf]# netstat -an | grep 9083
tcp 0 0 0.0.0.0:9083 0.0.0.0:* LISTEN
tcp 0 0 10.16.40.239:59522 10.16.40.239:9083 TIME_WAIT
tcp 0 0 10.16.40.239:59563 10.16.40.239:9083 TIME_WAIT
[root@ibm-x3650m4-02-vm-02 conf]#
If I start it via the UI, I get:
Python script has been killed due to timeout
But if I look in the hiveserver2.log I see this:
2014-10-28 12:53:13,615 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
2014-10-28 12:53:13,735 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
2014-10-28 12:53:13,954 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
2014-10-28 12:53:13,955 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(51)) - Initializing GlusterFS, CRC disabled.
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(52)) - GIT INFO={git.commit.id.abbrev=acca0e1, git.commit.user.email=bchilds@sh-el6.eng.rdu2.redhat.com, git.commit.message.full=[update RPM spec file/changelog] - 2.3.3
, git.commit.id=acca0e11de5f49a67474af70f6758d4182852760, git.commit.message.short=[update RPM spec file/changelog] - 2.3.3, git.commit.user.name=Brad Childs, git.build.user.name=Unknown, git.commit.id.describe=2.3.10-12-gacca0e1, git.build.user.email=Unknown, git.branch=2.3.3, git.commit.time=29.07.2014 @ 10:52:06 EDT, git.build.time=29.07.2014 @ 11:06:36 EDT}
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(53)) - GIT_TAG=2.3.10
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol1 at : /mnt/glusterfs/HadoopVol1
2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: gv0 at : /mnt/gv0
2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol2 at : /mnt/glusterfs/HadoopVol2
2014-10-28 12:53:14,053 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(155)) - Working directory is : glusterfs:/user/hive
2014-10-28 12:53:14,054 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(164)) - Write buffer size : 131072
2014-10-28 12:53:14,055 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(175)) - Default block size : 67108864
2014-10-28 12:53:14,056 INFO [main]: session.SessionState (SessionState.java:start(361)) - No Tez session required at this point. hive.execution.engine=mr.
2014-10-28 12:53:14,070 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(76)) - HiveServer2: Background operation thread pool size: 100
2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(78)) - HiveServer2: Background operation thread wait queue size: 100
2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(80)) - HiveServer2: Background operation thread keepalive time: 10
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
2014-10-28 12:53:14,130 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
2014-10-28 12:53:14,133 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
2014-10-28 12:53:14,209 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started.
2014-10-28 12:53:14,210 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
2014-10-28 12:53:14,399 INFO [Thread-4]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(97)) - ThriftBinaryCLIService listening on 0.0.0.0/0.0.0.0:10000
which makes it appear that it's working....
also this:
[root@ibm-x3650m4-02-vm-02 root]# ps -ef | grep hive
hcat 9251 1 0 Oct27 ? 00:02:31 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dwebhcat.log.dir=/var/log/webhcat/ -Dlog4j.configuration=file:///usr/lib/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hcat -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-0.13.0.2.1.7.0-784.jar org.apache.hive.hcatalog.templeton.Main
root 9808 10146 0 13:14 pts/1 00:00:10 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hadoop.hive.metastore.HiveMetaStore
root 20972 7782 0 13:32 pts/4 00:00:00 grep hive
hive 27185 1 0 12:49 ? 00:00:23 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris=" " -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
[root@ibm-x3650m4-02-vm-02 root]#
Hum........
But if you look at the UI, it still says stopped.
Erin
----- Original Message -----
From: "John Speidel" <js...@hortonworks.com>
To: dev@ambari.apache.org
Sent: Tuesday, October 28, 2014 10:33:12 AM
Subject: Re: Hive metastore cannot start
Erin,
Please provide an exported blueprint for your cluster so that we can take a
look at your configs.
http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint
Thanks,
John
On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <eb...@redhat.com> wrote:
> Hi,
> Has anyone else had an issue where the Hive Server and mysql will start
> but not the hive metastore?
> I am getting this in the hiveserver2.log logs:
>
> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
> at
> org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96)
> at
> org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:109)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
> ... 13 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
> ... 18 more
> Caused by: MetaException(message:Could not connect to meta store using any
> of the URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> This is running local (not remote). THe URI has been the FQDN, locallost,
> 0.0.0.0 -> yet I still get the same error.
> I have tried changing ports...
>
> Anyone seen this and knows how to fix/debug it?
>
> Erin
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hive metastore cannot start
Posted by Erin Boyd <eb...@redhat.com>.
HI John,
I also tried to start it via the command line
[root@ibm-x3650m4-02-vm-02 hive]# hive --service metastore
Starting Hive Metastore Server
Exception in thread "pool-3-thread-31" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-3-thread-33" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-3-thread-36" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:76)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
So from this I increased the heap size to 2048 (it was 1024).
It did however allow the listener to start:
[root@ibm-x3650m4-02-vm-02 conf]# netstat -an | grep 9083
tcp 0 0 0.0.0.0:9083 0.0.0.0:* LISTEN
tcp 0 0 10.16.40.239:59522 10.16.40.239:9083 TIME_WAIT
tcp 0 0 10.16.40.239:59563 10.16.40.239:9083 TIME_WAIT
[root@ibm-x3650m4-02-vm-02 conf]#
If I start it via the UI, I get:
Python script has been killed due to timeout
But if I look in the hiveserver2.log I see this:
2014-10-28 12:53:13,615 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
2014-10-28 12:53:13,735 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
2014-10-28 12:53:13,954 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
2014-10-28 12:53:13,955 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(51)) - Initializing GlusterFS, CRC disabled.
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(52)) - GIT INFO={git.commit.id.abbrev=acca0e1, git.commit.user.email=bchilds@sh-el6.eng.rdu2.redhat.com, git.commit.message.full=[update RPM spec file/changelog] - 2.3.3
, git.commit.id=acca0e11de5f49a67474af70f6758d4182852760, git.commit.message.short=[update RPM spec file/changelog] - 2.3.3, git.commit.user.name=Brad Childs, git.build.user.name=Unknown, git.commit.id.describe=2.3.10-12-gacca0e1, git.build.user.email=Unknown, git.branch=2.3.3, git.commit.time=29.07.2014 @ 10:52:06 EDT, git.build.time=29.07.2014 @ 11:06:36 EDT}
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:<init>(53)) - GIT_TAG=2.3.10
2014-10-28 12:53:13,957 INFO [main]: glusterfs.GlusterFileSystem (GlusterFileSystem.java:setConf(74)) - Configuring GlusterFS
2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(105)) - Initializing gluster volume..
2014-10-28 12:53:13,999 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol1 at : /mnt/glusterfs/HadoopVol1
2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: gv0 at : /mnt/gv0
2014-10-28 12:53:14,000 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(121)) - Gluster volume: HadoopVol2 at : /mnt/glusterfs/HadoopVol2
2014-10-28 12:53:14,053 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(155)) - Working directory is : glusterfs:/user/hive
2014-10-28 12:53:14,054 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(164)) - Write buffer size : 131072
2014-10-28 12:53:14,055 INFO [main]: glusterfs.GlusterVolume (GlusterVolume.java:setConf(175)) - Default block size : 67108864
2014-10-28 12:53:14,056 INFO [main]: session.SessionState (SessionState.java:start(361)) - No Tez session required at this point. hive.execution.engine=mr.
2014-10-28 12:53:14,070 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(76)) - HiveServer2: Background operation thread pool size: 100
2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(78)) - HiveServer2: Background operation thread wait queue size: 100
2014-10-28 12:53:14,071 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(80)) - HiveServer2: Background operation thread keepalive time: 10
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
2014-10-28 12:53:14,074 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
2014-10-28 12:53:14,075 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
2014-10-28 12:53:14,130 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://ibm-x3650m4-02-vm-02.lab.eng.bos.redhat.com:9083
2014-10-28 12:53:14,133 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore.
2014-10-28 12:53:14,209 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started.
2014-10-28 12:53:14,210 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
2014-10-28 12:53:14,399 INFO [Thread-4]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(97)) - ThriftBinaryCLIService listening on 0.0.0.0/0.0.0.0:10000
which makes it appear that it's working....
also this:
[root@ibm-x3650m4-02-vm-02 root]# ps -ef | grep hive
hcat 9251 1 0 Oct27 ? 00:02:31 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dwebhcat.log.dir=/var/log/webhcat/ -Dlog4j.configuration=file:///usr/lib/hive-hcatalog/sbin/../etc/webhcat/webhcat-log4j.properties -Dhadoop.log.dir=/var/log/hadoop/hcat -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hcat -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/hive-webhcat-0.13.0.2.1.7.0-784.jar org.apache.hive.hcatalog.templeton.Main
root 9808 10146 0 13:14 pts/1 00:00:10 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hadoop.hive.metastore.HiveMetaStore
root 20972 7782 0 13:32 pts/4 00:00:00 grep hive
hive 27185 1 0 12:49 ? 00:00:23 /usr/jdk64/jdk1.7.0_45/bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris=" " -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
[root@ibm-x3650m4-02-vm-02 root]#
Hum........
But if you look at the UI, it still says stopped.
Erin
----- Original Message -----
From: "John Speidel" <js...@hortonworks.com>
To: dev@ambari.apache.org
Sent: Tuesday, October 28, 2014 10:33:12 AM
Subject: Re: Hive metastore cannot start
Erin,
Please provide an exported blueprint for your cluster so that we can take a
look at your configs.
http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint
Thanks,
John
On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <eb...@redhat.com> wrote:
> Hi,
> Has anyone else had an issue where the Hive Server and mysql will start
> but not the hive metastore?
> I am getting this in the hiveserver2.log logs:
>
> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
> at
> org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96)
> at
> org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:109)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
> ... 13 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
> ... 18 more
> Caused by: MetaException(message:Could not connect to meta store using any
> of the URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> This is running local (not remote). THe URI has been the FQDN, locallost,
> 0.0.0.0 -> yet I still get the same error.
> I have tried changing ports...
>
> Anyone seen this and knows how to fix/debug it?
>
> Erin
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hive metastore cannot start
Posted by John Speidel <js...@hortonworks.com>.
Erin,
Please provide an exported blueprint for your cluster so that we can take a
look at your configs.
http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME?format=blueprint
Thanks,
John
On Tue, Oct 28, 2014 at 11:52 AM, Erin Boyd <eb...@redhat.com> wrote:
> Hi,
> Has anyone else had an issue where the Hive Server and mysql will start
> but not the hive metastore?
> I am getting this in the hiveserver2.log logs:
>
> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
> at
> org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:96)
> at
> org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:64)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:109)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
> ... 13 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
> ... 18 more
> Caused by: MetaException(message:Could not connect to meta store using any
> of the URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> This is running local (not remote). THe URI has been the FQDN, locallost,
> 0.0.0.0 -> yet I still get the same error.
> I have tried changing ports...
>
> Anyone seen this and knows how to fix/debug it?
>
> Erin
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.