[jira] Commented: (HBASE-1868) Spew about rebalancing but none done....

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759628#action_12759628 ] 

stack commented on HBASE-1868:
------------------------------

Looking.. but this seems to be at least part of why things were off:

{code}
2009-09-25 00:46:26,187 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 20020, call getClosestRowBefore([B@1c5fbf52, [B@5209af0b, [B@79c325
40) from XX.XX.XX.126:39465: error: java.io.IOException: Could not obtain block: blk_-4524474748365854543_61342811 file=/hbase/coral_hbase/.META./102878519
2/info/1652111193973935565
java.io.IOException: Could not obtain block: blk_-4524474748365854543_61342811 file=/hbase/coral_hbase/.META./1028785192/info/1652111193973935565
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1757)  
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)  
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
        at java.io.DataInputStream.read(Unknown Source)
        at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141)  
        at org.apache.hadoop.hbase.regionserver.Store.seekToScanner(Store.java:1134)   
        at org.apache.hadoop.hbase.regionserver.Store.rowAtOrBeforeFromStoreFile(Store.java:1101)
        at org.apache.hadoop.hbase.regionserver.Store.getRowKeyAtOrBefore(Store.java:1063)
        at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1036)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-09-25 00:46:26,224 DEBUG org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of info; new storefile is hdfs://coral-dfs.cluster.powe
rset.com:10000/hbase/coral_hbase/.META./1028785192/info/5419371614376079003; store size is 746.2k
{code}

It was complaining for a long while.. then got 'fixed' by the major compaction.

> Spew about rebalancing but none done....
> ----------------------------------------
>
>                 Key: HBASE-1868
>                 URL: https://issues.apache.org/jira/browse/HBASE-1868
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: 0.20.0 RC2
>            Reporter: stack
>
> I'm seeing loads of this in logs:
> {code}
> 2009-09-24 21:27:22,130 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server XX.XX.XX.100,20020,1253219583523 will be unloaded for balance. Server load: 5 avg: 3.78, regions can be moved: 4
> {code}
> Its like balancer is coming up w/ wrong answer to question... I don't see subsequent stuff going on... It does it over and over for hours.
> Then a split comes in and its seems to shake things up.  I see it do a bunch of assigning.
> {code}
> 2009-09-24 21:41:02,784 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_SPLIT: locations,,1253657949707: Daughters; locations,,1253
> 828460677, locations,http:\x2F\x2Fen.wikipedia.org\x2Fwiki\x2FLarry_Lucchino,1253828460677 from aa0-009-2.u.powerset.com,20020,1253219584971; 1 of 3
> 2009-09-24 21:41:02,784 DEBUG org.apache.hadoop.hbase.master.RegionManager: Assigning for address: XX.XX.XX.6:20020, startcode: 1253219584971, load: (reque
> sts=5213, regions=3, usedHeap=114, maxHeap=2031): total nregions to assign=2, nregions to reach balance=4, isMetaAssign=false
> 2009-09-24 21:41:02,820 DEBUG org.apache.hadoop.hbase.master.RegionManager: Assigning for address: XX.XX.XX.96:20020, startcode: 1253219584175, load: (requ
> ests=12, regions=4, usedHeap=404, maxHeap=2031): total nregions to assign=2, nregions to reach balance=4, isMetaAssign=false
> ...
> {code}
> Then back to the 'will be unloaded'... message.
> A new split comes in and then the assigning gets triggered again... a few regions are opened but not enough.
> Eventually it goes back to 'normal' (average load went to 3.85 from 3.8?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.