You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Stijn Diependaele <sd...@gmail.com> on 2012/04/03 10:03:51 UTC
Re: HBase starting problem
Sorry to dig up my thread again but couldn't my problem be related to this:
https://issues.apache.org/jira/browse/HADOOP-7988
?
since I have this error: GSS initiate failed [Caused by GSSException: No
valid credentials provided (Mechanism level: Server not found in Kerberos
database (7) - UNKNOWN_SERVER)] too.
2012/3/23 Sonal Goyal <so...@gmail.com>
> Your logs still point to a permissions issue on /hbase, so I would start at
> resolving that first thing..
>
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
> *Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
> KeeperErrorCode = AuthFailed for /hbase
> at org.apache.zookeeper.**KeeperException.creat*e(
> KeeperException.java:123)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Thu, Mar 22, 2012 at 6:56 PM, Stijn Diependaele <sdiepend@gmail.com
> >wrote:
>
> > Thanks for your reply,
> >
> > Currently I have no special permissions set for dfs, I think it's all
> > default.
> > It's the root that's running the dfs, same for zookeeper it's also the
> > root.
> > But for some reason the log from the hbase master is stored in Stijn,
> which
> > is a normal user, I'm not sure if this has anything to do with the
> problem.
> > Normally it should be root that runs it so it should be logged into the
> > hbase-root-master... file.
> >
> > logs/hbase-Stijn-master-HADOOP.domain.be.log
> >
> > Now I've removed the hbase.keytab and krb5.keytab and I made new ones.
> Now
> > I'm getting a different error:
> > And it seems that some problem could be the resolving of the hostname,
> but
> > normally that's configured well.
> >
> > .....
> > .....
> > 2012-03-22 14:19:28,166 INFO org.apache.zookeeper.ClientCnxn: Session
> > establishment complete on server HDP06.domain.be/10.11.12.206:2222,
> > sessionid = 0x1363a8fc39d0001, negotiated timeout = 180000
> > 2012-03-22 14:19:28,194 ERROR
> > org.apache.zookeeper.client.ZooKeeperSaslClient: An error:
> > (java.security.PrivilegedActionException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Server not
> > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when
> evaluating
> > Zookeeper Quorum Member's received SASL token. This may be caused by
> > Java's being unable to resolve the Zookeeper Quorum Member's hostname
> > correctly. You may want to try to adding
> > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> > environment. Zookeeper Client will go to AUTH_FAILED state.
> > 2012-03-22 14:19:28,194 ERROR org.apache.zookeeper.ClientCnxn: SASL
> > authentication with Zookeeper Quorum member failed:
> > javax.security.sasl.SaslException: An error:
> > (java.security.PrivilegedActionException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Server not
> > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when
> evaluating
> > Zookeeper Quorum Member's received SASL token. This may be caused by
> > Java's being unable to resolve the Zookeeper Quorum Member's hostname
> > correctly. You may want to try to adding
> > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> > environment. Zookeeper Client will go to AUTH_FAILED state.
> > 2012-03-22 14:19:28,229 ERROR
> > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> > java.lang.RuntimeException: Failed construction of Master: class
> > org.apache.hadoop.hbase.master.HMaster
> > at
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at
> >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
> > Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
> > KeeperErrorCode = AuthFailed for /hbase
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1031)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:189)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:892)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:159)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:152)
> > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:264)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> > at
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1583)
> > ... 5 more
> >
> >
> > 2012/3/21 Sonal Goyal <so...@gmail.com>
> >
> > > Hi Stijn,
> > >
> > > You seem to be hitting permission issues with /hbase
> > >
> > > Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
> > > KeeperErrorCode = InvalidACL for /hbase
> > > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
> > > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)
> > >
> > > What are your dfs user permission settings like? Which user are you
> > running
> > > hbase as?
> > >
> > > Best Regards,
> > > Sonal
> > > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > > Nube Technologies <http://www.nubetech.co>
> > >
> > > <http://in.linkedin.com/in/sonalgoyal>
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Mar 21, 2012 at 3:49 PM, Stijn Diependaele <sdiepend@gmail.com
> > > >wrote:
> > >
> > > > I'm trying to setup HBase on top of my HDFS filesystem.
> > > > I currently have 6 servers but only 3 are configured to run zookeeper
> > and
> > > > HBase, for testing purposes and the idea of adding the other servers
> > when
> > > > this works.
> > > > I'm following this guide
> > > http://hbase.apache.org/book/configuration.htmlto
> > > > set it up.
> > > >
> > > > I can paste the entire log in pastebin if you want.
> > > > HBase-site.xml : http://pastebin.com/Gb8UFSHj
> > > >
> > > > Errors in the log files:
> > > >
> > > > Zookeeper:
> > > > ----------
> > > >
> > > > logs/hbase-root-zookeeper-HADOOP.domain.be.log (master)
> > > > ....
> > > > ....
> > > > 2012-03-21 09:58:39,576 INFO
> > org.apache.zookeeper.server.ZooKeeperServer:
> > > > Established session 0x36347aae9a0000 with negotiated timeout 180000
> for
> > > > client /10.11.12.206:43325
> > > > 2012-03-21 09:58:39,611 WARN
> org.apache.zookeeper.server.NIOServerCnxn:
> > > > caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > > sessionid
> > > > 0x36347aae9a0000, likely client has closed socket
> > > > at
> > > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
> > > > at java.lang.Thread.run(Thread.java:722)
> > > > 2012-03-21 09:58:39,617 INFO
> org.apache.zookeeper.server.NIOServerCnxn:
> > > > Closed socket connection for client /10.11.12.206:43325 which had
> > > > sessionid
> > > > 0x36347aae9a0000
> > > > ....
> > > > ....
> > > >
> > > > on a znode:
> > > > logs/hbase-root-zookeeper-HDP08.domain.be.log
> > > > .....
> > > > .....
> > > > 2012-03-21 09:58:31,290 INFO
> > > > org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to
> > > majority
> > > > quorums
> > > > 2012-03-21 09:58:31,433 DEBUG
> org.apache.hadoop.hbase.util.VersionInfo:
> > > > preRegister called.
> > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908
> > > ,
> > > > name=log4j:logger=org.apache.hadoop.hbase.util.VersionInfo
> > > > 2012-03-21 09:58:31,434 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKConfig:
> > > > preRegister called.
> > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908
> > > ,
> > > > name=log4j:logger=org.apache.hadoop.hbase.zookeeper.ZKConfig
> > > > 2012-03-21 09:58:31,434 DEBUG
> > org.apache.hadoop.hbase.HBaseConfiguration:
> > > > preRegister called.
> > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908
> > > ,
> > > > name=log4j:logger=org.apache.hadoop.hbase.HBaseConfiguration
> > > > 2012-03-21 09:58:31,435 DEBUG org.apache.hadoop.hbase: preRegister
> > > called.
> > > > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
> > > > name=log4j:logger=org.apache.hadoop.hbase
> > > > 2012-03-21 09:58:31,435 INFO
> > > > org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum
> peer
> > > > 2012-03-21 09:58:31,836 INFO org.apache.zookeeper.Login: successfully
> > > > logged in.
> > > > 2012-03-21 09:58:31,842 INFO org.apache.zookeeper.Login: TGT refresh
> > > thread
> > > > started.
> > > > .....
> > > > .....
> > > > a bit further down the log file:
> > > > ....
> > > > ....
> > > > 2012-03-21 09:58:32,425 INFO
> > > > org.apache.zookeeper.server.quorum.LearnerHandler: Sending snapshot
> > last
> > > > zxid of peer is 0x1800000012 zxid of leader is 0x1900000000sent zxid
> > of
> > > db
> > > > as 0x1800000012
> > > > 2012-03-21 09:58:32,466 WARN
> org.apache.zookeeper.server.quorum.Leader:
> > > > Commiting zxid 0x1900000000 from /10.11.12.208:2888 not first!
> > > > 2012-03-21 09:58:32,466 WARN
> org.apache.zookeeper.server.quorum.Leader:
> > > > First is 0
> > > > 2012-03-21 09:58:32,466 INFO
> org.apache.zookeeper.server.quorum.Leader:
> > > > Have quorum of supporters; starting up and setting last processed
> zxid:
> > > > 107374182400
> > > > 2012-03-21 09:58:36,505 INFO
> > > > org.apache.zookeeper.server.quorum.QuorumCnxManager: Received
> > connection
> > > > request /10.11.12.200:46015
> > > > ....
> > > > ....
> > > >
> > > > HBase:
> > > > -----
> > > > master:
> > > > ....
> > > > ....
> > > > 2012-03-21 09:58:39,162 WARN org.apache.zookeeper.ClientCnxn:
> Zookeeper
> > > > client cannot authenticate using the Client section of the supplied
> > > > configuration file: '/home/Stijn/hbase-0.92.0/conf/jaas.conf'. Will
> > > > continue connection to Zookeeper server without SASL authentication,
> if
> > > > Zookeeper server allows it.
> > > > ....
> > > > ....
> > > > 2012-03-21 09:58:39,313 INFO org.apache.zookeeper.ClientCnxn: Session
> > > > establishment complete on server HDP06.domain.be/10.11.12.206:2222,
> > > > sessionid = 0x136347a9db40000, negotiated timeout = 180000
> > > > 2012-03-21 09:58:39,454 ERROR
> > > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start
> > master
> > > > java.lang.RuntimeException: Failed construction of Master: class
> > > > org.apache.hadoop.hbase.master.HMaster
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> > > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
> > > > Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
> > > > KeeperErrorCode = InvalidACL for /hbase
> > > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
> > > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)
> > > > .....
> > > > .....
> > > >
> > >
> >
>