You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Zhong, Andy" <Sh...@searshc.com> on 2011/07/15 06:43:01 UTC
RE: Hadoop/HBase Upgrade from 0.20.3 to 0.90.2
St.Ack,
It's weird. I did Hbase upgrade from Hbase 0.20.3 to Hbase 0.90.2 by following "replace the Hadoop JAR file from Hadoop 0.20.2 release with the sync-supporting Hadoop JAR file found in HBase 0.90.2". It was suceesfully before, but now we are facing error from Hbase shell: NativeException: org.apache.hadoop.hbase.MasterNotRunningException: hofvmecln15.ecom.sears.com:60000, even though the processes of Hmaster and Region are running, and 60010 can be accessible via 60010.
Could you please help to find out the possible root cause? Below is the logging messages from the hbase.log from Master Node:
2011-07-14 22:54:55,903 INFO zookeeper.ZooKeeper - Client environment:user.dir=/appl/hbase/share/hbase-0.90.2
2011-07-14 22:54:55,904 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=hofvmecln19:2181,hofvmecln18:2181,hofvmecln17:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@51b4a74b
2011-07-14 22:54:55,905 INFO zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false
2011-07-14 22:54:55,932 INFO zookeeper.ClientCnxn - Attempting connection to server hofvmecln18/151.149.116.46:2181
2011-07-14 22:54:55,938 INFO zookeeper.ClientCnxn - Priming connection to java.nio.channels.SocketChannel[connected local=/151.149.119.93:52156 remote=hofvmecln18/151.149.116.46:2181]
2011-07-14 22:54:55,939 INFO zookeeper.ClientCnxn - Server connection successful
2011-07-14 22:54:56,030 INFO regionserver.HRegionServer - Got ZooKeeper event, state: SyncConnected, type: None, path: null
2011-07-14 22:54:56,202 INFO regionserver.HRegionServer - Telling master at hofvmecln15.ecom.sears.com:60000 that we are up
2011-07-14 22:54:56,407 WARN ipc.HBaseServer - IPC Server listener on 60000: readAndProcess threw exception java.io.EOFException. Count of bytes read: 0
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readUTF(DataInputStream.java:592)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:122)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:978)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:946)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2011-07-14 22:54:56,412 WARN regionserver.HRegionServer - Unable to connect to master. Retrying. Error was:
java.io.IOException: Call to hofvmecln15.ecom.sears.com/151.149.119.93:60000 failed on local exception: java.io.EOFException
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:757)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:727)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:423)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1296)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)
thanks,
Andy Zhong
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Wednesday, June 08, 2011 11:53 AM
To: user@hbase.apache.org
Subject: Re: Hadoop/HBase Upgrade Suggestion
On Wed, Jun 8, 2011 at 9:25 AM, Zhong, Sheng <Sh...@searshc.com> wrote:
> Could anyone give me suggestion for Hadoop/HBase upgrade? We're
> currently using apache hadoop 0.20.2 + hbase 0.20.3 + zookeeper-3.2.2.
> Has anyone done with latest stable version of hadoop-0.20.203.0rc1 +
> Hbase 0.90.2, and will Hbase 0.90.2 have compatible issue with
> hadoop-0.20.203.0rc1? I am very appreciated by any help and suggestion.
>
hadoop-0.20.203 does not have append support so its not recommended that you run hbase on it. Here is our (a little stale at this point
-- see recent thread) story regards hadoop versions http://hbase.apache.org/book/notsoquick.html#hadoop
St.Ack
This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
Re: Hadoop/HBase Upgrade from 0.20.3 to 0.90.2
Posted by Joey Echeverria <jo...@cloudera.com>.
Hey Andy,
This looks like the log from the regions server. Any chance you can post the
log for the HMaster?
-Joey
On Fri, Jul 15, 2011 at 12:43 AM, Zhong, Andy <Sh...@searshc.com>wrote:
> St.Ack,
>
> It's weird. I did Hbase upgrade from Hbase 0.20.3 to Hbase 0.90.2 by
> following "replace the Hadoop JAR file from Hadoop 0.20.2 release with the
> sync-supporting Hadoop JAR file found in HBase 0.90.2". It was suceesfully
> before, but now we are facing error from Hbase shell: NativeException:
> org.apache.hadoop.hbase.MasterNotRunningException:
> hofvmecln15.ecom.sears.com:60000, even though the processes of Hmaster and
> Region are running, and 60010 can be accessible via 60010.
>
> Could you please help to find out the possible root cause? Below is the
> logging messages from the hbase.log from Master Node:
>
> 2011-07-14 22:54:55,903 INFO zookeeper.ZooKeeper - Client
> environment:user.dir=/appl/hbase/share/hbase-0.90.2
> 2011-07-14 22:54:55,904 INFO zookeeper.ZooKeeper - Initiating client
> connection, connectString=hofvmecln19:2181,hofvmecln18:2181,hofvmecln17:2181
> sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@51b4a74b
> 2011-07-14 22:54:55,905 INFO zookeeper.ClientCnxn -
> zookeeper.disableAutoWatchReset is false
> 2011-07-14 22:54:55,932 INFO zookeeper.ClientCnxn - Attempting connection
> to server hofvmecln18/151.149.116.46:2181
> 2011-07-14 22:54:55,938 INFO zookeeper.ClientCnxn - Priming connection to
> java.nio.channels.SocketChannel[connected local=/151.149.119.93:52156remote=hofvmecln18/
> 151.149.116.46:2181]
> 2011-07-14 22:54:55,939 INFO zookeeper.ClientCnxn - Server connection
> successful
> 2011-07-14 22:54:56,030 INFO regionserver.HRegionServer - Got ZooKeeper
> event, state: SyncConnected, type: None, path: null
> 2011-07-14 22:54:56,202 INFO regionserver.HRegionServer - Telling master
> at hofvmecln15.ecom.sears.com:60000 that we are up
> 2011-07-14 22:54:56,407 WARN ipc.HBaseServer - IPC Server listener on
> 60000: readAndProcess threw exception java.io.EOFException. Count of bytes
> read: 0
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readUTF(DataInputStream.java:592)
> at java.io.DataInputStream.readUTF(DataInputStream.java:547)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:122)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:978)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:946)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-07-14 22:54:56,412 WARN regionserver.HRegionServer - Unable to
> connect to master. Retrying. Error was:
> java.io.IOException: Call to
> hofvmecln15.ecom.sears.com/151.149.119.93:60000 failed on local exception:
> java.io.EOFException
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:757)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:727)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
> at $Proxy0.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:423)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1296)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)
>
>
> thanks,
> Andy Zhong
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Wednesday, June 08, 2011 11:53 AM
> To: user@hbase.apache.org
> Subject: Re: Hadoop/HBase Upgrade Suggestion
>
> On Wed, Jun 8, 2011 at 9:25 AM, Zhong, Sheng <Sh...@searshc.com>
> wrote:
> > Could anyone give me suggestion for Hadoop/HBase upgrade? We're
> > currently using apache hadoop 0.20.2 + hbase 0.20.3 + zookeeper-3.2.2.
> > Has anyone done with latest stable version of hadoop-0.20.203.0rc1 +
> > Hbase 0.90.2, and will Hbase 0.90.2 have compatible issue with
> > hadoop-0.20.203.0rc1? I am very appreciated by any help and suggestion.
> >
>
> hadoop-0.20.203 does not have append support so its not recommended that
> you run hbase on it. Here is our (a little stale at this point
> -- see recent thread) story regards hadoop versions
> http://hbase.apache.org/book/notsoquick.html#hadoop
>
> St.Ack
>
> This message, including any attachments, is the property of Sears Holdings
> Corporation and/or one of its subsidiaries. It is confidential and may
> contain proprietary or legally privileged information. If you are not the
> intended recipient, please delete it without reading the contents. Thank
> you.
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434