You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Daniel Ploeg <dp...@gmail.com> on 2008/09/19 01:49:49 UTC

Unable to get to data

Hi all,

I downloaded the latest release candidate (0.18.0 - with hadoop 0.18.0) to
evaluate whether it would suit my purposes and to get more familiar with the
software. I'm running on a cluster with 1 master and 4 region servers.

I was running some simple tests yesterday, inserting records into a table
and reading them out (largely based on the example in the API).

Today when I went to access the the table I was getting some exceptions. The
following was from my little test app that reads / writes to HBase:

org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
to locate root region
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:789)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:432)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:442)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:446)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:407)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:86)

I also tried to do a describe of the table using the shell and this was the
result:

NativeException: java.lang.NullPointerException: null
    from java/lang/String.java:433:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:75:in `toString'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
`getMessage'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
`<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:863:in
`getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:297:in
`listTables'
    from org/apache/hadoop/hbase/client/HBaseAdmin.java:117:in `listTables'
    from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
    from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
    from java/lang/reflect/Method.java:585:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
`invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
... 133 levels...
    from
ruby.data1.home.fast.tools.hbase.hbase_minus_0_dot_18_dot_0.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
`call'
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:351:in
`__file__'
    from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
`__file__'
    from
ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
`load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /home/fast/tools/hbase/current/bin/../bin/hirb.rb:246:in `describe'

I can see the data on HDFS through the browser, but for some reason I can't
get to it.

A short time later I noticed that my region processes had died (after I had
run the describe). I went back through the logs and noticed the following
repeated error that started yesterday:

2008-09-18 16:03:34,834 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
Rolling hlog. Number of entries: 51
2008-09-18 16:03:34,834 ERROR
org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
java.io.IOException: Could not get block locations. Aborting...
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)

then finally:

2008-09-19 09:28:02,477 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
java.io.IOException: java.io.IOException: java.lang.NullPointerException
        at java.lang.String.<init>(String.java:433)
        at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:75)
        at
org.apache.hadoop.hbase.client.RetriesExhaustedException.getMessage(RetriesExhaustedException.java:50)
        at
org.apache.hadoop.hbase.client.RetriesExhaustedException.<init>(RetriesExhaustedException.java:40)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
        at
org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:237)
        at
org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:216)
        at
org.apache.hadoop.hbase.RegionHistorian.addRegionAssignment(RegionHistorian.java:140)
        at
org.apache.hadoop.hbase.master.RegionManager.assignRegionsToMultipleServers(RegionManager.java:263)
        at
org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:199)
        at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:367)
        at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:313)
        at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:232)
        at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:564)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at
java.lang.reflect.Constructor.newInstance(Constructor.java:494)        at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:399)
        at java.lang.Thread.run(Thread.java:595)
2008-09-19 09:28:02,478 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
master for 1993183 milliseconds - aborting server

A restart of hbase does not seem to have resolved the issue either. I get
the following error on the webpage when I try to access my table:

HTTP ERROR: 500

java.io.IOException: Could not obtain block:
blk_-1560243342111364142_23058
file=/hbase/-ROOT-/70236052/info/mapfiles/293988224693195769/data
	at org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
	at org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
	at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
	at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1369)
	at java.io.DataInputStream.readInt(DataInputStream.java:353)
	at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1895)
	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1925)
	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
	at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516)
	at org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1504)
	at org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1476)
	at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1250)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1057)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)


Could someone please advise as to what might be the problem and how to
resolve it.

Thanks,
Daniel

Re: Unable to get to data

Posted by Daniel Ploeg <dp...@gmail.com>.
Yes I believe so. The properties in hadoop that have been overridden
include:

fs.default.name
mapred.job.tracker
dfs.name.dir
dfs.data.dir
mapred.local.dir
mapred.child.java.opts
keep.failed.task.files

I haven't overriden hadoop.tmp.dir, but I can still see some data in there.
Should I change this?

Is there anything I might have missed?

Thanks,
Daniel


On Fri, Sep 19, 2008 at 11:21 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:

> Daniel,
>
> Many of the errors I see seem related to the fact that something went wrong
> under the hood... But got an NPE will trying to process it (seems the
> region
> name was null hence no region found).
>
> It seems to indicate that your data in HDFS went away. Have you changed
> Hadoop default config that puts the data into /tmp ?
>
> J-D
>
> On Thu, Sep 18, 2008 at 7:49 PM, Daniel Ploeg <dp...@gmail.com> wrote:
>
> > Hi all,
> >
> > I downloaded the latest release candidate (0.18.0 - with hadoop 0.18.0)
> to
> > evaluate whether it would suit my purposes and to get more familiar with
> > the
> > software. I'm running on a cluster with 1 master and 4 region servers.
> >
> > I was running some simple tests yesterday, inserting records into a table
> > and reading them out (largely based on the example in the API).
> >
> > Today when I went to access the the table I was getting some exceptions.
> > The
> > following was from my little test app that reads / writes to HBase:
> >
> > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> trying
> > to locate root region
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:789)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:432)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:442)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:446)
> >    at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:407)
> >    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101)
> >    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:86)
> >
> > I also tried to do a describe of the table using the shell and this was
> the
> > result:
> >
> > NativeException: java.lang.NullPointerException: null
> >    from java/lang/String.java:433:in `<init>'
> >    from org/apache/hadoop/hbase/util/Bytes.java:75:in `toString'
> >    from
> org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
> > `getMessage'
> >    from
> org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
> > `<init>'
> >    from org/apache/hadoop/hbase/client/HConnectionManager.java:863:in
> > `getRegionServerWithRetries'
> >    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
> >    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
> >    from org/apache/hadoop/hbase/client/HConnectionManager.java:297:in
> > `listTables'
> >    from org/apache/hadoop/hbase/client/HBaseAdmin.java:117:in
> `listTables'
> >    from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
> >    from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
> >    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >    from java/lang/reflect/Method.java:585:in `invoke'
> >    from org/jruby/javasupport/JavaMethod.java:250:in
> > `invokeWithExceptionHandling'
> >    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> >    from org/jruby/javasupport/JavaClass.java:416:in `execute'
> > ... 133 levels...
> >    from
> >
> >
> ruby.data1.home.fast.tools.hbase.hbase_minus_0_dot_18_dot_0.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
> > `call'
> >    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> `call'
> >    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> `call'
> >    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> >    from org/jruby/runtime/CallSite.java:298:in `call'
> >    from
> >
> >
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:351:in
> > `__file__'
> >    from
> >
> >
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
> > `__file__'
> >    from
> >
> >
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
> > `load'
> >    from org/jruby/Ruby.java:512:in `runScript'
> >    from org/jruby/Ruby.java:432:in `runNormally'
> >    from org/jruby/Ruby.java:312:in `runFromMain'
> >    from org/jruby/Main.java:144:in `run'
> >    from org/jruby/Main.java:89:in `run'
> >    from org/jruby/Main.java:80:in `main'
> >    from /home/fast/tools/hbase/current/bin/../bin/hirb.rb:246:in
> `describe'
> >
> > I can see the data on HDFS through the browser, but for some reason I
> can't
> > get to it.
> >
> > A short time later I noticed that my region processes had died (after I
> had
> > run the describe). I went back through the logs and noticed the following
> > repeated error that started yesterday:
> >
> > 2008-09-18 16:03:34,834 INFO
> > org.apache.hadoop.hbase.regionserver.LogRoller:
> > Rolling hlog. Number of entries: 51
> > 2008-09-18 16:03:34,834 ERROR
> > org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> > java.io.IOException: Could not get block locations. Aborting...
> >        at
> >
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
> >        at
> >
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
> >        at
> >
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
> >
> > then finally:
> >
> > 2008-09-19 09:28:02,477 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> > (Retry: 0)
> > java.io.IOException: java.io.IOException: java.lang.NullPointerException
> >        at java.lang.String.<init>(String.java:433)
> >        at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:75)
> >        at
> >
> >
> org.apache.hadoop.hbase.client.RetriesExhaustedException.getMessage(RetriesExhaustedException.java:50)
> >        at
> >
> >
> org.apache.hadoop.hbase.client.RetriesExhaustedException.<init>(RetriesExhaustedException.java:40)
> >        at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
> >        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
> >        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
> >        at
> > org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:237)
> >        at
> > org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:216)
> >        at
> >
> >
> org.apache.hadoop.hbase.RegionHistorian.addRegionAssignment(RegionHistorian.java:140)
> >        at
> >
> >
> org.apache.hadoop.hbase.master.RegionManager.assignRegionsToMultipleServers(RegionManager.java:263)
> >        at
> >
> >
> org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:199)
> >        at
> >
> >
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:367)
> >        at
> >
> >
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:313)
> >        at
> >
> >
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:232)
> >        at
> >
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:564)
> >        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> >        at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:585)
> >        at
> > org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> >
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >        at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >        at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >        at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:494)        at
> >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> >        at
> >
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> >        at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:399)
> >        at java.lang.Thread.run(Thread.java:595)
> > 2008-09-19 09:28:02,478 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 1993183 milliseconds - aborting server
> >
> > A restart of hbase does not seem to have resolved the issue either. I get
> > the following error on the webpage when I try to access my table:
> >
> > HTTP ERROR: 500
> >
> > java.io.IOException: Could not obtain block:
> > blk_-1560243342111364142_23058
> > file=/hbase/-ROOT-/70236052/info/mapfiles/293988224693195769/data
> >        at
> >
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
> >        at
> >
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
> >        at
> > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
> >        at
> > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1369)
> >        at java.io.DataInputStream.readInt(DataInputStream.java:353)
> >        at
> >
> org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1895)
> >        at
> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1925)
> >        at
> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
> >        at
> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> >        at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1504)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1476)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1250)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1057)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:585)
> >        at
> > org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> >
> >
> > Could someone please advise as to what might be the problem and how to
> > resolve it.
> >
> > Thanks,
> > Daniel
> >
>

Re: Unable to get to data

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Daniel,

Many of the errors I see seem related to the fact that something went wrong
under the hood... But got an NPE will trying to process it (seems the region
name was null hence no region found).

It seems to indicate that your data in HDFS went away. Have you changed
Hadoop default config that puts the data into /tmp ?

J-D

On Thu, Sep 18, 2008 at 7:49 PM, Daniel Ploeg <dp...@gmail.com> wrote:

> Hi all,
>
> I downloaded the latest release candidate (0.18.0 - with hadoop 0.18.0) to
> evaluate whether it would suit my purposes and to get more familiar with
> the
> software. I'm running on a cluster with 1 master and 4 region servers.
>
> I was running some simple tests yesterday, inserting records into a table
> and reading them out (largely based on the example in the API).
>
> Today when I went to access the the table I was getting some exceptions.
> The
> following was from my little test app that reads / writes to HBase:
>
> org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
> to locate root region
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:789)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:432)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:442)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:414)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:551)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:446)
>    at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:407)
>    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101)
>    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:86)
>
> I also tried to do a describe of the table using the shell and this was the
> result:
>
> NativeException: java.lang.NullPointerException: null
>    from java/lang/String.java:433:in `<init>'
>    from org/apache/hadoop/hbase/util/Bytes.java:75:in `toString'
>    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
> `getMessage'
>    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
> `<init>'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:863:in
> `getRegionServerWithRetries'
>    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
>    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:297:in
> `listTables'
>    from org/apache/hadoop/hbase/client/HBaseAdmin.java:117:in `listTables'
>    from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>    from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>    from java/lang/reflect/Method.java:585:in `invoke'
>    from org/jruby/javasupport/JavaMethod.java:250:in
> `invokeWithExceptionHandling'
>    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
>    from org/jruby/javasupport/JavaClass.java:416:in `execute'
> ... 133 levels...
>    from
>
> ruby.data1.home.fast.tools.hbase.hbase_minus_0_dot_18_dot_0.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
> `call'
>    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
>    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
>    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
>    from org/jruby/runtime/CallSite.java:298:in `call'
>    from
>
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:351:in
> `__file__'
>    from
>
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
> `__file__'
>    from
>
> ruby/data1/home/fast/tools/hbase/hbase_minus_0_dot_18_dot_0/bin//home/fast/tools/hbase/current/bin/../bin/hirb.rb:-1:in
> `load'
>    from org/jruby/Ruby.java:512:in `runScript'
>    from org/jruby/Ruby.java:432:in `runNormally'
>    from org/jruby/Ruby.java:312:in `runFromMain'
>    from org/jruby/Main.java:144:in `run'
>    from org/jruby/Main.java:89:in `run'
>    from org/jruby/Main.java:80:in `main'
>    from /home/fast/tools/hbase/current/bin/../bin/hirb.rb:246:in `describe'
>
> I can see the data on HDFS through the browser, but for some reason I can't
> get to it.
>
> A short time later I noticed that my region processes had died (after I had
> run the describe). I went back through the logs and noticed the following
> repeated error that started yesterday:
>
> 2008-09-18 16:03:34,834 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> Rolling hlog. Number of entries: 51
> 2008-09-18 16:03:34,834 ERROR
> org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.io.IOException: Could not get block locations. Aborting...
>        at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
>        at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
>        at
>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
>
> then finally:
>
> 2008-09-19 09:28:02,477 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
>        at java.lang.String.<init>(String.java:433)
>        at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:75)
>        at
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException.getMessage(RetriesExhaustedException.java:50)
>        at
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException.<init>(RetriesExhaustedException.java:40)
>        at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
>        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
>        at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
>        at
> org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:237)
>        at
> org.apache.hadoop.hbase.RegionHistorian.add(RegionHistorian.java:216)
>        at
>
> org.apache.hadoop.hbase.RegionHistorian.addRegionAssignment(RegionHistorian.java:140)
>        at
>
> org.apache.hadoop.hbase.master.RegionManager.assignRegionsToMultipleServers(RegionManager.java:263)
>        at
>
> org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:199)
>        at
>
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:367)
>        at
>
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:313)
>        at
>
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:232)
>        at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:564)
>        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>        at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at
> java.lang.reflect.Constructor.newInstance(Constructor.java:494)        at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
>        at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:399)
>        at java.lang.Thread.run(Thread.java:595)
> 2008-09-19 09:28:02,478 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 1993183 milliseconds - aborting server
>
> A restart of hbase does not seem to have resolved the issue either. I get
> the following error on the webpage when I try to access my table:
>
> HTTP ERROR: 500
>
> java.io.IOException: Could not obtain block:
> blk_-1560243342111364142_23058
> file=/hbase/-ROOT-/70236052/info/mapfiles/293988224693195769/data
>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
>        at
> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1369)
>        at java.io.DataInputStream.readInt(DataInputStream.java:353)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1895)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1925)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
>        at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516)
>        at
> org.apache.hadoop.hbase.regionserver.HStore.rowAtOrBeforeFromMapFile(HStore.java:1504)
>        at
> org.apache.hadoop.hbase.regionserver.HStore.getRowKeyAtOrBefore(HStore.java:1476)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1250)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1057)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>
>
> Could someone please advise as to what might be the problem and how to
> resolve it.
>
> Thanks,
> Daniel
>