You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ved Prakash <me...@gmail.com> on 2008/03/14 06:42:42 UTC
Regionserver error while uploading large data file
Hi,
I am trying to upload close to 4 GB of data on hbase, but I am not able to
do. Following are my observations:
1. Before failing number of rows inserted were about 800,000, and the time
taken for this upload was close to 6 hours, and this is just 1/10th of total
rows that I have to insert,. This way it would take me ages before I can
insert all the data into the table. And by the time I would be finished I
would have similar size ready for another insertion. Is there any better way
to do this?
2. I saw this happens whenever a datanode goes down. Doesn't hbase have
mechanism where it continues loading data to other datanodes, in case one
fails?
For inserting data in the table, I wrote a php script having database
connectivity using the REST [ https://issues.apache.org/jira/browse/HBASE-37]
Since the datafile is big, and on my first execution I faced a similar
situation even when none of my datanodes were down. I thought of breaking it
into smaller files and then processing them, but, it doesn't help.
Can you guys help me with this.
Thanks
-------------------------------------------------
hbase-hadoop-
Re: Regionserver error while uploading large data file
Posted by stack <st...@duboce.net>.
Ved Prakash wrote:
> Hi,
>
> I am trying to upload close to 4 GB of data on hbase, but I am not able to
> do. Following are my observations:
>
> 1. Before failing number of rows inserted were about 800,000, and the time
> taken for this upload was close to 6 hours, and this is just 1/10th of total
> rows that I have to insert,. This way it would take me ages before I can
> insert all the data into the table. And by the time I would be finished I
> would have similar size ready for another insertion. Is there any better way
> to do this?
>
Tell us more about your setup Ved. How many regionservers? What
version of hbase? Is your uploader single-threaded (If so, you need to
fix this)?
> 2. I saw this happens whenever a datanode goes down. Doesn't hbase have
> mechanism where it continues loading data to other datanodes, in case one
> fails?
>
It does. Regions from failed server get deployed elsewhere and away we
go again (at least, thats how its supposed to work in theory).
> For inserting data in the table, I wrote a php script having database
> connectivity using the REST [ https://issues.apache.org/jira/browse/HBASE-37]
> Since the datafile is big, and on my first execution I faced a similar
> situation even when none of my datanodes were down. I thought of breaking it
> into smaller files and then processing them, but, it doesn't help.
>
Tell us also more about your data format and schema. What size are the
inserts? Are you inserting rows in lexicographically ascending order
(not so good) or are your inserts to random locations in the row
namespace (better)?
Its probably smart breaking up a big file into smaller pieces, unless
you have a means of resuming from where uploads failed.
St.Ack
> Can you guys help me with this.
>
> Thanks
> -------------------------------------------------
>
> hbase-hadoop-
>
>
Re: Regionserver error while uploading large data file
Posted by stack <st...@duboce.net>.
Ved:
The REST client failed because:
java.net.SocketTimeoutException: timed out waiting for rpc
The master is trying to talk to the regionserver that was hosting the
catalog .META. table but its getting:
java.io.IOException: Server not running
It looks like the regionserver that was hosting the .META. is not
running or crashed. Look in its logs to see if you can figure why it
went away.
St.Ack
Ved Prakash wrote:
> hbase-root-rest-ved-desktop.log.2008-03-13
>
> 2008-03-13 14:30:46,067 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.Server@133f1d7
> 2008-03-13 20:06:47,136 INFO org.mortbay.util.Container: Started
> HttpContext[/,/]
> 2008-03-13 23:37:12,047 WARN rest: /api/CreativeViewDetails/row/4102047/:
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy0.findRootRegion(Unknown Source)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRootRegion(
> HConnectionManager.java:700)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
> HConnectionManager.java:329)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
> HConnectionManager.java:306)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
> HConnectionManager.java:402)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
> HConnectionManager.java:339)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
> HConnectionManager.java:311)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
> HConnectionManager.java:476)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
> HConnectionManager.java:343)
> at
> org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
> HConnectionManager.java:311)
> at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:113)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:977)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:941)
> at org.apache.hadoop.hbase.rest.TableHandler.putRowXml(
> TableHandler.java:335)
> at org.apache.hadoop.hbase.rest.TableHandler.putRow(
> TableHandler.java:254)
> at org.apache.hadoop.hbase.rest.TableHandler.doPost(
> TableHandler.java:89)
> at org.apache.hadoop.hbase.rest.Dispatcher.doPost(Dispatcher.java
> :113)
> at org.apache.hadoop.hbase.rest.Dispatcher.doPut(Dispatcher.java
> :126)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:712)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java
> :427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
> WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(
> ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(
> WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java
> :981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(
> SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> response
> at org.apache.hadoop.ipc.Client.call(Client.java:514)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java
> :210)
> ... 33 more
> ------------------------------------------------------
> error listing from stdout where hbase master was running
> (Partial list)
> 08/03/14 10:37:33 WARN ipc.Server: IPC Server Responder, call
> next(-6267545063026670769) from 192.168.0.52:35791: output error
> 08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 6 on 39677 caught:
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
> SocketChannelImpl.java:125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
> at org.apache.hadoop.ipc.Server$Responder.processResponse(
> Server.java:608)
> at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:666)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:930)
>
> 08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 3 on 39677, call
> openScanner(.META.,,1, [Lorg.apache.hadoop.io.Text;@e66f56, , 1205471252608,
> null) from 192.168.0.52:35817: error: java.io.IOException: Server not
> running
> java.io.IOException: Server not running
> at org.apache.hadoop.hbase.HRegionServer.checkOpen(
> HRegionServer.java:1626)
> at org.apache.hadoop.hbase.HRegionServer.openScanner(
> HRegionServer.java:1448)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
> :413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 08/03/14 10:37:33 WARN hbase.HMaster: Scan one META region: {regionname:
> .META.,,1, startKey: <>, server: 192.168.0.52:39677}
> java.io.IOException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.HRegionServer.checkOpen(
> HRegionServer.java:1626)
> at org.apache.hadoop.hbase.HRegionServer.openScanner(
> HRegionServer.java:1448)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
> :413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(
> RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(
> HMaster.java:259)
> at org.apache.hadoop.hbase.HMaster$MetaScanner.scanOneMetaRegion(
> HMaster.java:687)
> at org.apache.hadoop.hbase.HMaster$MetaScanner.maintenanceScan(
> HMaster.java:743)
> at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java
> :201)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:63)
> 08/03/14 10:37:33 INFO hbase.HMaster: all meta regions scanned
> 08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 2 on 39677, call
> getClosestRowBefore(.META.,,1, CreativeViewDetails,4111546,999999999999999,
> 9223372036854775807) from 192.168.0.52:35850: error: java.io.IOException:
> Server not running
> java.io.IOException: Server not running
> at org.apache.hadoop.hbase.HRegionServer.checkOpen(
> HRegionServer.java:1626)
> at org.apache.hadoop.hbase.HRegionServer.getClosestRowBefore(
> HRegionServer.java:1363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
> :413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 08/03/14 10:37:36 INFO hbase.HRegionServer: RegionServer:0.cacheFlusherexiting
> 08/03/14 10:37:40 INFO hbase.HRegionServer: worker thread exiting
> 08/03/14 10:37:42 INFO hbase.Leases: RegionServer:0 2 lease(s) outstanding.
> Waiting for them to expire.
> 08/03/14 10:37:47 ERROR hbase.HRegionServer: Error opening scanner (fsOk:
> true)
> java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
> HStore.java:2172)
> at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java
> :2293)
> at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2083)
> at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java
> :1622)
> at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1196)
> at org.apache.hadoop.hbase.HRegionServer.openScanner(
> HRegionServer.java:1453)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
> :413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> response
> at org.apache.hadoop.ipc.Client.call(Client.java:514)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
> at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:444)
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:334)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
> :1387)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
> :1382)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:263)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
> at org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader
> .<init>(HStoreFile.java:593)
> at org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader
> .<init>(HStoreFile.java:648)
> at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:419)
> at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
> HStore.java:2147)
> ... 11 more
> I hope this helps
>
>
Re: Regionserver error while uploading large data file
Posted by Ved Prakash <me...@gmail.com>.
hbase-root-rest-ved-desktop.log.2008-03-13
>>>>
2008-03-13 14:30:46,067 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.Server@133f1d7
2008-03-13 20:06:47,136 INFO org.mortbay.util.Container: Started
HttpContext[/,/]
2008-03-13 23:37:12,047 WARN rest: /api/CreativeViewDetails/row/4102047/:
java.lang.reflect.UndeclaredThrowableException
at $Proxy0.findRootRegion(Unknown Source)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRootRegion(
HConnectionManager.java:700)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:329)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:306)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
HConnectionManager.java:402)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:339)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
HConnectionManager.java:311)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
HConnectionManager.java:476)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:343)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
HConnectionManager.java:311)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:113)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:977)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:941)
at org.apache.hadoop.hbase.rest.TableHandler.putRowXml(
TableHandler.java:335)
at org.apache.hadoop.hbase.rest.TableHandler.putRow(
TableHandler.java:254)
at org.apache.hadoop.hbase.rest.TableHandler.doPost(
TableHandler.java:89)
at org.apache.hadoop.hbase.rest.Dispatcher.doPost(Dispatcher.java
:113)
at org.apache.hadoop.hbase.rest.Dispatcher.doPut(Dispatcher.java
:126)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:712)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java
:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(
ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(
WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java
:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(
SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:514)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java
:210)
... 33 more
------------------------------------------------------
error listing from stdout where hbase master was running
(Partial list)
08/03/14 10:37:33 WARN ipc.Server: IPC Server Responder, call
next(-6267545063026670769) from 192.168.0.52:35791: output error
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 6 on 39677 caught:
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.Server$Responder.processResponse(
Server.java:608)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:930)
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 3 on 39677, call
openScanner(.META.,,1, [Lorg.apache.hadoop.io.Text;@e66f56, , 1205471252608,
null) from 192.168.0.52:35817: error: java.io.IOException: Server not
running
java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
08/03/14 10:37:33 WARN hbase.HMaster: Scan one META region: {regionname:
.META.,,1, startKey: <>, server: 192.168.0.52:39677}
java.io.IOException: java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(
RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(
HMaster.java:259)
at org.apache.hadoop.hbase.HMaster$MetaScanner.scanOneMetaRegion(
HMaster.java:687)
at org.apache.hadoop.hbase.HMaster$MetaScanner.maintenanceScan(
HMaster.java:743)
at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java
:201)
at org.apache.hadoop.hbase.Chore.run(Chore.java:63)
08/03/14 10:37:33 INFO hbase.HMaster: all meta regions scanned
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 2 on 39677, call
getClosestRowBefore(.META.,,1, CreativeViewDetails,4111546,999999999999999,
9223372036854775807) from 192.168.0.52:35850: error: java.io.IOException:
Server not running
java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.getClosestRowBefore(
HRegionServer.java:1363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
08/03/14 10:37:36 INFO hbase.HRegionServer: RegionServer:0.cacheFlusherexiting
08/03/14 10:37:40 INFO hbase.HRegionServer: worker thread exiting
08/03/14 10:37:42 INFO hbase.Leases: RegionServer:0 2 lease(s) outstanding.
Waiting for them to expire.
08/03/14 10:37:47 ERROR hbase.HRegionServer: Error opening scanner (fsOk:
true)
java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
HStore.java:2172)
at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java
:2293)
at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2083)
at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java
:1622)
at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1196)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1453)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:514)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:444)
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(
DistributedFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1387)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1382)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:263)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
at org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader
.<init>(HStoreFile.java:593)
at org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader
.<init>(HStoreFile.java:648)
at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:419)
at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
HStore.java:2147)
... 11 more
I hope this helps