You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Weihua JIANG <we...@gmail.com> on 2011/07/04 10:57:39 UTC
Region not online after split by a closing RS
Hi all,
We encountered a problem about region not onlining. A region is
splitted by a closing RS and then this RS down. It seems master has
known this split but it doesn't tried to make it online. Log from
master
2011-06-30 22:58:52,945 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
and split region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
checking daughter presence
2011-06-30 22:58:52,946 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop01.sh.intel.com,50820,1309421825940,
region=ed60ec735e30db1d99290995eb1cd2d7
2011-06-30 22:58:53,005 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
present
2011-06-30 22:58:53,065 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
present
Log from RS is:
2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 73 on 50820 caught:
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 73 on 50820: exiting
2011-06-30 22:57:05,767 INFO
org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
leases
2011-06-30 22:57:05,768 INFO
org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
leases
2011-06-30 22:57:05,768 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x130ba69074900b4
2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
0x130ba69074900b4 closed
2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
2011-06-30 22:57:05,857 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
2011-06-30 22:57:05,863 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
2011-06-30 22:57:05,911 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
in META
2011-06-30 22:57:05,942 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
in region .META.,,1, serverInfo=null
2011-06-30 22:57:05,943 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
because stopping=false, stopped=true
2011-06-30 22:57:05,950 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
in region .META.,,1, serverInfo=null
2011-06-30 22:57:05,950 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
because stopping=false, stopped=true
2011-06-30 22:57:06,004 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
new regions: CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
Split took 1mins, 12sec
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Split Thread to finish...
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Large Compaction Thread to finish...
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Small Compaction Thread to finish...
2011-06-30 22:57:06,004 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
exiting
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting; hbase.shutdown.hook=true;
fsShutdownHook=Thread[Thread-15,5,main]
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
hook
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
shutdown hook thread.
2011-06-30 22:57:06,196 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
finished.
Thanks
Weihua
Re: Region not online after split by a closing RS
Posted by Ted Yu <yu...@gmail.com>.
I filed https://issues.apache.org/jira/browse/HBASE-4060
Feel free to fill out detail there.
On Mon, Jul 4, 2011 at 7:55 PM, Ted Yu <yu...@gmail.com> wrote:
> The root cause was that ServerShutdownHandler.isDaughterMissing() uses
> FindDaughterVisitor which doesn't check whether the value for
> HConstants.SERVER_QUALIFIER carries non-null value.
> Therefore for the scenario below isDaughterMissing() returns false,
> skipping the following line:
> assignmentManager.assign(daughter, true);
>
> Please file a JIRA.
>
> Thanks for reporting this issue.
>
>
>
Re: Region not online after split by a closing RS
Posted by Weihua JIANG <we...@gmail.com>.
HBASE-4059 https://issues.apache.org/jira/browse/HBASE-4059 is created.
Thanks
Weihua
2011/7/5 Ted Yu <yu...@gmail.com>:
> The root cause was that ServerShutdownHandler.isDaughterMissing() uses
> FindDaughterVisitor which doesn't check whether the value for
> HConstants.SERVER_QUALIFIER carries non-null value.
> Therefore for the scenario below isDaughterMissing() returns false, skipping
> the following line:
> assignmentManager.assign(daughter, true);
>
> Please file a JIRA.
>
> Thanks for reporting this issue.
>
> On Mon, Jul 4, 2011 at 7:10 PM, Weihua JIANG <we...@gmail.com> wrote:
>
>> According to RS log, it seems it performs a region split during this
>> RS shutdown process. Thus, RS only performs the split and add the two
>> daughter regions to META, but not open them (MetaEditor.addDaughter()
>> with serverInfo as null). It seems it's the correct behavior for RS.
>>
>> But, I can't find the corresponding code on master side to assign
>> these unassigned regions. And the log on master side doesn't show an
>> hint of such assignment decision.
>>
>> Thanks
>> Weihua
>>
>> 2011/7/5 Weihua JIANG <we...@gmail.com>:
>> > Both daughter regions are offline.
>> >
>> > The full log of master related to this RS down are:
>> >
>> > 2011-06-30 22:56:51,621 INFO
>> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
>> > ephemeral node deleted, processing expiration
>> > [xmlqa-clv4.sh.intel.com,50820,1309421874765]
>> > 2011-06-30 22:56:51,622 DEBUG
>> > org.apache.hadoop.hbase.master.ServerManager:
>> > Added=xmlqa-clv4.sh.intel.com,50820,1309421874765 to dead servers,
>> > submitted shutdown handler to be executed, root=false, meta=false
>> > 2011-06-30 22:56:51,622 INFO
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
>> > Splitting logs for xmlqa-clv4.sh.intel.com,50820,1309421874765
>> > 2011-06-30 22:56:51,633 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 35
>> > hlog(s) in hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
>> > 2011-06-30 22:56:51,633 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
>> > Thread[WriterThread-0,5,main]: starting
>> > 2011-06-30 22:56:51,633 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
>> > Thread[WriterThread-1,5,main]: starting
>> > 2011-06-30 22:56:51,633 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 1 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> ,
>> > length=63870691
>> > 2011-06-30 22:56:51,634 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> > 2011-06-30 22:56:51,634 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
>> > Thread[WriterThread-2,5,main]: starting
>> > 2011-06-30 22:56:52,636 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> > 2011-06-30 22:56:52,736 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,736 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> > 2011-06-30 22:56:52,777 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,777 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
>> > region=625da13f7c9f8d2a6db3e7bd892de061
>> > 2011-06-30 22:56:52,790 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,790 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
>> > region=6a22ac864ca20a7f34b8822b6b092697
>> > 2011-06-30 22:56:52,825 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,826 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
>> > region=0ea99d0d995b95e8ea07441d30aa5545
>> > 2011-06-30 22:56:52,836 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,836 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
>> > region=32c8d35a95c6ac4f399c17dfa04066b1
>> > 2011-06-30 22:56:52,841 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,841 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
>> > region=d5a18e344d77bdccb568fd59623fb566
>> > 2011-06-30 22:56:52,868 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,868 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
>> > region=e4a2320bc8ff30fee1a9b20267b399cf
>> > 2011-06-30 22:56:52,909 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:52,909 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
>> > region=de5cb72653d016804cbd16f4a71470cd
>> > 2011-06-30 22:56:53,017 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,018 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
>> > region=5a4a923e81c6127eb7f3ed61e5774063
>> > 2011-06-30 22:56:53,023 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,024 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
>> > region=3e0c81217d4b47920e11960f1816efae
>> > 2011-06-30 22:56:53,069 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,069 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
>> > region=d673806c5348d8517dcdb3392e5f5b21
>> > 2011-06-30 22:56:53,101 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,101 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
>> > region=7cfa5c93f2d46d01c8f60ff934053ad1
>> > 2011-06-30 22:56:53,109 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,109 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
>> > region=84c6f6ce13ffbeb3eb83c765fef754f3
>> > 2011-06-30 22:56:53,179 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,179 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
>> > region=c94dc395183fd5a2286ec823b194b517
>> > 2011-06-30 22:56:53,190 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
>> > syncFs -- HDFS-200
>> > 2011-06-30 22:56:53,190 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
>> > path=hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
>> > region=0dd9c11896eca61505861a9e49f2fef3
>> > 2011-06-30 22:56:54,198 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25752
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> > 2011-06-30 22:56:54,198 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 2 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> ,
>> > length=73429706
>> > 2011-06-30 22:56:54,198 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> > 2011-06-30 22:56:55,201 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> > 2011-06-30 22:56:57,787 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=29756
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> > 2011-06-30 22:56:57,787 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 3 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> ,
>> > length=65063774
>> > 2011-06-30 22:56:57,788 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> > 2011-06-30 22:56:58,793 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> > 2011-06-30 22:57:00,533 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26415
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> > 2011-06-30 22:57:00,533 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 4 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> ,
>> > length=64631565
>> > 2011-06-30 22:57:00,533 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> > 2011-06-30 22:57:01,537 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> > 2011-06-30 22:57:03,755 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25941
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> > 2011-06-30 22:57:03,755 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 5 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> ,
>> > length=68367834
>> > 2011-06-30 22:57:03,755 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> > 2011-06-30 22:57:04,757 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> > 2011-06-30 22:57:06,463 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27679
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> > 2011-06-30 22:57:06,463 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 6 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> ,
>> > length=64503295
>> > 2011-06-30 22:57:06,463 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> > 2011-06-30 22:57:07,465 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> > 2011-06-30 22:57:09,089 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26171
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> > 2011-06-30 22:57:09,089 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 7 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> ,
>> > length=64201147
>> > 2011-06-30 22:57:09,089 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> > 2011-06-30 22:57:10,093 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> > 2011-06-30 22:57:11,792 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25904
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> > 2011-06-30 22:57:11,792 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 8 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> ,
>> > length=63800579
>> > 2011-06-30 22:57:11,792 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> > 2011-06-30 22:57:12,794 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> > 2011-06-30 22:57:18,504 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> > 2011-06-30 22:57:18,504 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 9 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> ,
>> > length=64119848
>> > 2011-06-30 22:57:18,504 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> > 2011-06-30 22:57:19,506 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> > 2011-06-30 22:57:21,160 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26023
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> > 2011-06-30 22:57:21,160 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 10 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> ,
>> > length=64643463
>> > 2011-06-30 22:57:21,160 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> > 2011-06-30 22:57:22,162 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> > 2011-06-30 22:57:23,541 DEBUG
>> > org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
>> > currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
>> > dead server
>> > 2011-06-30 22:57:23,573 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server listener on 50800: readAndProcess threw exception
>> > java.io.IOException: Connection reset by peer. Count of bytes read: 0
>> > java.io.IOException: Connection reset by peer
>> > at sun.nio.ch.FileDispatcher.read0(Native Method)
>> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> > at sun.nio.ch.IOUtil.read(IOUtil.java:206)
>> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1357)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>> > at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:619)
>> > 2011-06-30 22:57:23,674 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26324
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> > 2011-06-30 22:57:23,674 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 11 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> ,
>> > length=64232645
>> > 2011-06-30 22:57:23,674 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> > 2011-06-30 22:57:23,734 DEBUG
>> > org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
>> > currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
>> > dead server
>> > 2011-06-30 22:57:24,678 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> > 2011-06-30 22:57:26,287 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25901
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> > 2011-06-30 22:57:26,288 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 12 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> ,
>> > length=68916014
>> > 2011-06-30 22:57:26,288 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> > 2011-06-30 22:57:27,291 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> > 2011-06-30 22:57:29,540 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27997
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> > 2011-06-30 22:57:29,540 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 13 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> ,
>> > length=64023210
>> > 2011-06-30 22:57:29,541 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> > 2011-06-30 22:57:30,543 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> > 2011-06-30 22:57:32,118 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25852
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> > 2011-06-30 22:57:32,118 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 14 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> ,
>> > length=63963259
>> > 2011-06-30 22:57:32,118 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> > 2011-06-30 22:57:33,123 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> > 2011-06-30 22:57:36,027 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25800
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> > 2011-06-30 22:57:36,027 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 15 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> ,
>> > length=65390175
>> > 2011-06-30 22:57:36,028 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> > 2011-06-30 22:57:37,031 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> > 2011-06-30 22:57:38,618 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26591
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> > 2011-06-30 22:57:38,618 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 16 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> ,
>> > length=64082403
>> > 2011-06-30 22:57:38,618 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> > 2011-06-30 22:57:39,623 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> > 2011-06-30 22:57:42,129 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26079
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> > 2011-06-30 22:57:42,129 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 17 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> ,
>> > length=64438939
>> > 2011-06-30 22:57:42,129 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> > 2011-06-30 22:57:43,132 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> > 2011-06-30 22:57:44,965 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26012
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> > 2011-06-30 22:57:44,965 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 18 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> ,
>> > length=64295447
>> > 2011-06-30 22:57:44,965 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> > 2011-06-30 22:57:45,968 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> > 2011-06-30 22:57:48,826 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26038
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> > 2011-06-30 22:57:48,826 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 19 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> ,
>> > length=64592159
>> > 2011-06-30 22:57:48,826 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> > 2011-06-30 22:57:49,828 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> > 2011-06-30 22:57:53,219 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26061
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> > 2011-06-30 22:57:53,219 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 20 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> ,
>> > length=64057243
>> > 2011-06-30 22:57:53,220 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> > 2011-06-30 22:57:54,224 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> > 2011-06-30 22:57:58,220 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25765
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> > 2011-06-30 22:57:58,220 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 21 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> ,
>> > length=64306648
>> > 2011-06-30 22:57:58,220 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> > 2011-06-30 22:57:59,225 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> > 2011-06-30 22:58:00,572 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26006
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> > 2011-06-30 22:58:00,572 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 22 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> ,
>> > length=64238208
>> > 2011-06-30 22:58:00,572 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> > 2011-06-30 22:58:01,577 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> > 2011-06-30 22:58:04,010 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25961
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> > 2011-06-30 22:58:04,010 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 23 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> ,
>> > length=66022861
>> > 2011-06-30 22:58:04,011 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> > 2011-06-30 22:58:05,017 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> > 2011-06-30 22:58:08,583 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26599
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> > 2011-06-30 22:58:08,583 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 24 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> ,
>> > length=64341194
>> > 2011-06-30 22:58:08,583 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> > 2011-06-30 22:58:09,585 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> > 2011-06-30 22:58:11,171 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25978
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> > 2011-06-30 22:58:11,171 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 25 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> ,
>> > length=64083581
>> > 2011-06-30 22:58:11,171 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> > 2011-06-30 22:58:12,173 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> > 2011-06-30 22:58:14,655 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26044
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> > 2011-06-30 22:58:14,656 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 26 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> ,
>> > length=64093785
>> > 2011-06-30 22:58:14,656 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> > 2011-06-30 22:58:15,658 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> > 2011-06-30 22:58:17,914 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25712
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> > 2011-06-30 22:58:17,915 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 27 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> ,
>> > length=64115777
>> > 2011-06-30 22:58:17,915 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> > 2011-06-30 22:58:18,918 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> > 2011-06-30 22:58:20,762 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> > 2011-06-30 22:58:20,762 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 28 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> ,
>> > length=64628500
>> > 2011-06-30 22:58:20,762 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> > 2011-06-30 22:58:21,766 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> > 2011-06-30 22:58:23,158 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26131
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> > 2011-06-30 22:58:23,159 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 29 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> ,
>> > length=64522869
>> > 2011-06-30 22:58:23,159 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> > 2011-06-30 22:58:24,162 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> > 2011-06-30 22:58:25,593 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26159
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> > 2011-06-30 22:58:25,593 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 30 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> ,
>> > length=64127204
>> > 2011-06-30 22:58:25,593 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> > 2011-06-30 22:58:26,594 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> > 2011-06-30 22:58:28,023 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> > 2011-06-30 22:58:28,023 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 31 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> ,
>> > length=96405599
>> > 2011-06-30 22:58:28,023 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> > 2011-06-30 22:58:29,026 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> > 2011-06-30 22:58:31,603 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38922
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> > 2011-06-30 22:58:31,603 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 32 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> ,
>> > length=64219650
>> > 2011-06-30 22:58:31,603 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> > 2011-06-30 22:58:32,607 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> > 2011-06-30 22:58:33,924 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> > 2011-06-30 22:58:33,924 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 33 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> ,
>> > length=64734596
>> > 2011-06-30 22:58:33,924 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> > 2011-06-30 22:58:34,927 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> > 2011-06-30 22:58:37,455 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26330
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> > 2011-06-30 22:58:37,456 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 34 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> ,
>> > length=95982267
>> > 2011-06-30 22:58:37,456 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> > 2011-06-30 22:58:38,459 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> > 2011-06-30 22:58:40,216 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38754
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> > 2011-06-30 22:58:40,216 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
>> > 35 of 35: hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> ,
>> > length=0
>> > 2011-06-30 22:58:40,216 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Recovering file
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > 2011-06-30 22:58:41,219 INFO org.apache.hadoop.hbase.util.FSUtils:
>> > Finished lease recover attempt for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > 2011-06-30 22:58:41,219 WARN
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > might be still open, length is 0
>> > 2011-06-30 22:58:42,146 DEBUG
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=23462
>> > entries from hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > 2011-06-30 22:58:42,179 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
>> > 2011-06-30 22:58:42,185 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
>> > 2011-06-30 22:58:42,191 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
>> > 2011-06-30 22:58:42,196 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
>> > 2011-06-30 22:58:42,203 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
>> > 2011-06-30 22:58:42,255 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
>> > 2011-06-30 22:58:42,270 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
>> > 2011-06-30 22:58:42,276 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
>> > 2011-06-30 22:58:42,292 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
>> > 2011-06-30 22:58:42,300 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
>> > 2011-06-30 22:58:42,306 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
>> > 2011-06-30 22:58:42,321 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
>> > 2011-06-30 22:58:42,329 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
>> > 2011-06-30 22:58:42,336 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
>> > 2011-06-30 22:58:42,342 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
>> > 2011-06-30 22:58:42,348 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
>> > 2011-06-30 22:58:42,354 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
>> > 2011-06-30 22:58:42,360 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
>> > 2011-06-30 22:58:42,366 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
>> > 2011-06-30 22:58:42,372 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
>> > 2011-06-30 22:58:42,378 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
>> > 2011-06-30 22:58:42,408 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
>> > 2011-06-30 22:58:42,414 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
>> > 2011-06-30 22:58:42,420 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
>> > 2011-06-30 22:58:42,426 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
>> > 2011-06-30 22:58:42,437 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
>> > 2011-06-30 22:58:42,444 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
>> > 2011-06-30 22:58:42,491 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
>> > 2011-06-30 22:58:42,504 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
>> > 2011-06-30 22:58:42,510 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
>> > 2011-06-30 22:58:42,516 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
>> > 2011-06-30 22:58:42,522 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
>> > 2011-06-30 22:58:42,528 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
>> > 2011-06-30 22:58:42,534 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
>> > 2011-06-30 22:58:42,540 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
>> > processed log hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > to hdfs://
>> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
>> > 2011-06-30 22:58:42,546 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for
>> > split writer threads to finish
>> > 2011-06-30 22:58:49,942 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers
>> > finished
>> > 2011-06-30 22:58:50,002 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
>> > (wrote 62511 edits in 6859ms)
>> > 2011-06-30 22:58:50,014 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
>> > (wrote 63273 edits in 9130ms)
>> > 2011-06-30 22:58:51,226 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
>> > (wrote 63193 edits in 29376ms)
>> > 2011-06-30 22:58:51,232 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
>> > (wrote 62727 edits in 16595ms)
>> > 2011-06-30 22:58:52,413 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
>> > (wrote 62672 edits in 7177ms)
>> > 2011-06-30 22:58:52,461 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
>> > (wrote 62636 edits in 52110ms)
>> > 2011-06-30 22:58:52,509 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
>> > (wrote 62860 edits in 19402ms)
>> > 2011-06-30 22:58:52,515 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
>> > (wrote 62718 edits in 12592ms)
>> > 2011-06-30 22:58:52,605 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
>> > (wrote 62561 edits in 12576ms)
>> > 2011-06-30 22:58:52,670 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
>> > (wrote 62860 edits in 9056ms)
>> > 2011-06-30 22:58:52,719 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
>> > (wrote 63328 edits in 8381ms)
>> > 2011-06-30 22:58:52,725 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
>> > (wrote 62040 edits in 7900ms)
>> > 2011-06-30 22:58:52,731 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
>> > (wrote 62397 edits in 19988ms)
>> > 2011-06-30 22:58:52,779 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
>> > (wrote 62952 edits in 21391ms)
>> > 2011-06-30 22:58:52,785 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
>> > (wrote 62765 edits in 15781ms)
>> > 2011-06-30 22:58:52,785 INFO
>> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
>> > splitting completed in 121163 ms for
>> > hdfs://
>> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
>> > 2011-06-30 22:58:52,893 INFO
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
>> > Reassigning 16 region(s) that
>> > xmlqa-clv4.sh.intel.com,50820,1309421874765 was carrying (skipping 0
>> > regions(s) that are already in transition)
>> > 2011-06-30 22:58:52,894 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 6a22ac864ca20a7f34b8822b6b092697 with OFFLINE state
>> > 2011-06-30 22:58:52,896 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.,
>> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:52,896 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
>> > to hadoop05.sh.intel.com,50820,1309421836726
>> > 2011-06-30 22:58:52,898 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 625da13f7c9f8d2a6db3e7bd892de061 with OFFLINE state
>> > 2011-06-30 22:58:52,900 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.,
>> > src=, dest=harpertown083.sh.intel.com,50820,1309421853805; 10
>> > (online=10, exclude=null) available servers
>> > 2011-06-30 22:58:52,900 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
>> > to harpertown083.sh.intel.com,50820,1309421853805
>> > 2011-06-30 22:58:52,900 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop05.sh.intel.com,50820,1309421836726,
>> > region=6a22ac864ca20a7f34b8822b6b092697
>> > 2011-06-30 22:58:52,902 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 32c8d35a95c6ac4f399c17dfa04066b1 with OFFLINE state
>> > 2011-06-30 22:58:52,904 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.,
>> > src=, dest=xmlqa-clv6.sh.intel.com,50820,1309421880784; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:52,904 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
>> > to xmlqa-clv6.sh.intel.com,50820,1309421880784
>> > 2011-06-30 22:58:52,904 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=harpertown083.sh.intel.com,50820,1309421853805,
>> > region=625da13f7c9f8d2a6db3e7bd892de061
>> > 2011-06-30 22:58:52,906 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 0ea99d0d995b95e8ea07441d30aa5545 with OFFLINE state
>> > 2011-06-30 22:58:52,935 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.,
>> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:52,935 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
>> > to hadoop05.sh.intel.com,50820,1309421836726
>> > 2011-06-30 22:58:52,936 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=xmlqa-clv6.sh.intel.com,50820,1309421880784,
>> > region=32c8d35a95c6ac4f399c17dfa04066b1
>> > 2011-06-30 22:58:52,937 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for d5a18e344d77bdccb568fd59623fb566 with OFFLINE state
>> > 2011-06-30 22:58:52,939 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.,
>> > src=, dest=hbase01.sh.intel.com,50820,1309421885586; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:52,939 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
>> > to hbase01.sh.intel.com,50820,1309421885586
>> > 2011-06-30 22:58:52,940 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop05.sh.intel.com,50820,1309421836726,
>> > region=0ea99d0d995b95e8ea07441d30aa5545
>> > 2011-06-30 22:58:52,941 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for ed60ec735e30db1d99290995eb1cd2d7 with OFFLINE state
>> > 2011-06-30 22:58:52,943 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hbase01.sh.intel.com,50820,1309421885586,
>> > region=d5a18e344d77bdccb568fd59623fb566
>> > 2011-06-30 22:58:52,943 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.,
>> > src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:52,943 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
>> > to hadoop01.sh.intel.com,50820,1309421825940
>> > 2011-06-30 22:58:52,945 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>> > and split region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>> > checking daughter presence
>> > 2011-06-30 22:58:52,946 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> > 2011-06-30 22:58:53,005 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > present
>> > 2011-06-30 22:58:53,065 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > present
>> > 2011-06-30 22:58:53,065 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 3e0c81217d4b47920e11960f1816efae with OFFLINE state
>> > 2011-06-30 22:58:53,067 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.,
>> > src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,067 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
>> > to hadoop01.sh.intel.com,50820,1309421825940
>> > 2011-06-30 22:58:53,070 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for e4a2320bc8ff30fee1a9b20267b399cf with OFFLINE state
>> > 2011-06-30 22:58:53,072 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> > region=3e0c81217d4b47920e11960f1816efae
>> > 2011-06-30 22:58:53,072 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.,
>> > src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,072 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
>> > to hadoop06.sh.intel.com,50820,1309421846110
>> > 2011-06-30 22:58:53,083 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 5a4a923e81c6127eb7f3ed61e5774063 with OFFLINE state
>> > 2011-06-30 22:58:53,085 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.,
>> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,085 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
>> > to ditest02.sh.intel.com,50820,1309421803646
>> > 2011-06-30 22:58:53,085 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop06.sh.intel.com,50820,1309421846110,
>> > region=e4a2320bc8ff30fee1a9b20267b399cf
>> > 2011-06-30 22:58:53,086 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for d673806c5348d8517dcdb3392e5f5b21 with OFFLINE state
>> > 2011-06-30 22:58:53,087 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.,
>> > src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,087 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
>> > to hadoop06.sh.intel.com,50820,1309421846110
>> > 2011-06-30 22:58:53,088 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=ditest02.sh.intel.com,50820,1309421803646,
>> > region=5a4a923e81c6127eb7f3ed61e5774063
>> > 2011-06-30 22:58:53,090 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 7cfa5c93f2d46d01c8f60ff934053ad1 with OFFLINE state
>> > 2011-06-30 22:58:53,117 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.,
>> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,117 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
>> > to hadoop05.sh.intel.com,50820,1309421836726
>> > 2011-06-30 22:58:53,117 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop06.sh.intel.com,50820,1309421846110,
>> > region=d673806c5348d8517dcdb3392e5f5b21
>> > 2011-06-30 22:58:53,120 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 84c6f6ce13ffbeb3eb83c765fef754f3 with OFFLINE state
>> > 2011-06-30 22:58:53,126 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.,
>> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,126 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
>> > to ditest02.sh.intel.com,50820,1309421803646
>> > 2011-06-30 22:58:53,127 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop05.sh.intel.com,50820,1309421836726,
>> > region=7cfa5c93f2d46d01c8f60ff934053ad1
>> > 2011-06-30 22:58:53,129 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for c94dc395183fd5a2286ec823b194b517 with OFFLINE state
>> > 2011-06-30 22:58:53,143 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.,
>> > src=, dest=ditest01.sh.intel.com,50820,1309421799506; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,143 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
>> > to ditest01.sh.intel.com,50820,1309421799506
>> > 2011-06-30 22:58:53,143 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=ditest02.sh.intel.com,50820,1309421803646,
>> > region=84c6f6ce13ffbeb3eb83c765fef754f3
>> > 2011-06-30 22:58:53,149 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=ditest01.sh.intel.com,50820,1309421799506,
>> > region=c94dc395183fd5a2286ec823b194b517
>> > 2011-06-30 22:58:53,151 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for 0dd9c11896eca61505861a9e49f2fef3 with OFFLINE state
>> > 2011-06-30 22:58:53,153 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.,
>> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
>> > exclude=null) available servers
>> > 2011-06-30 22:58:53,153 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
>> > to ditest02.sh.intel.com,50820,1309421803646
>> > 2011-06-30 22:58:53,156 DEBUG
>> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
>> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
>> > for d9adac25d8b83a3d4c252acaf5ba17af with OFFLINE state
>> > 2011-06-30 22:58:53,157 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
>> > transition plan was found (or we are ignoring an existing plan) for
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
>> > so generated a random one;
>> >
>> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.,
>> > src=, dest=harpertown0816.sh.intel.com,50820,1309421864349; 10
>> > (online=10, exclude=null) available servers
>> > 2011-06-30 22:58:53,157 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
>> > to harpertown0816.sh.intel.com,50820,1309421864349
>> > 2011-06-30 22:58:53,158 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=ditest02.sh.intel.com,50820,1309421803646,
>> > region=0dd9c11896eca61505861a9e49f2fef3
>> > 2011-06-30 22:58:53,159 INFO
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
>> > processing of shutdown of xmlqa-clv4.sh.intel.com,50820,1309421874765
>> >
>> >
>> > Thanks
>> > Weihua
>> >
>> > 2011/7/4 Ted Yu <yu...@gmail.com>:
>> >> In the future, please direct questions on cdh releases to
>> >> cdh-dev@cloudera.org
>> >> You may cc dev@hbase.apache.org
>> >>
>> >> There is more than one minute difference between master and RS logs.
>> >> Which one of the daughter regions didn't come online ?
>> >>
>> >> Cheers
>> >>
>> >> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com>
>> wrote:
>> >>
>> >>> The HBase version we are using is CDH3U0.
>> >>>
>> >>> Thanks
>> >>> Weihua
>> >>>
>> >>> 2011/7/4 Weihua JIANG <we...@gmail.com>:
>> >>> > Hi all,
>> >>> >
>> >>> > We encountered a problem about region not onlining. A region is
>> >>> > splitted by a closing RS and then this RS down. It seems master has
>> >>> > known this split but it doesn't tried to make it online. Log from
>> >>> > master
>> >>> > 2011-06-30 22:58:52,945 DEBUG
>> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
>> Offlined
>> >>> > and split region
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>> >>> > checking daughter presence
>> >>> > 2011-06-30 22:58:52,946 DEBUG
>> >>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> >>> > transition=RS_ZK_REGION_OPENING,
>> >>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> >>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> >>> > 2011-06-30 22:58:53,005 DEBUG
>> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
>> Daughter
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> >>> > present
>> >>> > 2011-06-30 22:58:53,065 DEBUG
>> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
>> Daughter
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> >>> > present
>> >>> >
>> >>> > Log from RS is:
>> >>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> >>> > Server handler 73 on 50820 caught:
>> >>> > java.nio.channels.ClosedChannelException
>> >>> > at
>> >>>
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> >>> > at
>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> >>> > at
>> >>>
>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>> >>> > at
>> >>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> >>> > at
>> >>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> >>> > at
>> >>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>> >>> >
>> >>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> >>> > Server handler 73 on 50820: exiting
>> >>> > 2011-06-30 22:57:05,767 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820
>> closing
>> >>> > leases
>> >>> > 2011-06-30 22:57:05,768 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>> >>> > leases
>> >>> > 2011-06-30 22:57:05,768 INFO
>> >>> >
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >>> > Closed zookeeper sessionid=0x130ba69074900b4
>> >>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> >>> > 0x130ba69074900b4 closed
>> >>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>> >>> > EventThread shut down
>> >>> > 2011-06-30 22:57:05,857 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> >>> > 2011-06-30 22:57:05,863 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> >>> > 2011-06-30 22:57:05,911 INFO
>> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>> >>> > in META
>> >>> > 2011-06-30 22:57:05,942 INFO
>> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> >>> > in region .META.,,1, serverInfo=null
>> >>> > 2011-06-30 22:57:05,943 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> >>> > daughter
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> >>> > because stopping=false, stopped=true
>> >>> > 2011-06-30 22:57:05,950 INFO
>> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> >>> > in region .META.,,1, serverInfo=null
>> >>> > 2011-06-30 22:57:05,950 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> >>> > daughter
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> >>> > because stopping=false, stopped=true
>> >>> > 2011-06-30 22:57:06,004 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>> >>> > updated, and report to master.
>> >>> >
>> >>>
>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>> >>> > new regions:
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>> >>> >
>> >>>
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>> >>> > Split took 1mins, 12sec
>> >>> > 2011-06-30 22:57:06,004 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> >>> > Split Thread to finish...
>> >>> > 2011-06-30 22:57:06,004 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> >>> > Large Compaction Thread to finish...
>> >>> > 2011-06-30 22:57:06,004 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> >>> > Small Compaction Thread to finish...
>> >>> > 2011-06-30 22:57:06,004 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>> >>> > exiting
>> >>> > 2011-06-30 22:57:06,090 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >>> > starting; hbase.shutdown.hook=true;
>> >>> > fsShutdownHook=Thread[Thread-15,5,main]
>> >>> > 2011-06-30 22:57:06,090 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> >>> > hook
>> >>> > 2011-06-30 22:57:06,090 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> >>> > shutdown hook thread.
>> >>> > 2011-06-30 22:57:06,196 INFO
>> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >>> > finished.
>> >>> >
>> >>> >
>> >>> > Thanks
>> >>> > Weihua
>> >>> >
>> >>>
>> >>
>> >
>>
>
Re: Region not online after split by a closing RS
Posted by Ted Yu <yu...@gmail.com>.
The root cause was that ServerShutdownHandler.isDaughterMissing() uses
FindDaughterVisitor which doesn't check whether the value for
HConstants.SERVER_QUALIFIER carries non-null value.
Therefore for the scenario below isDaughterMissing() returns false, skipping
the following line:
assignmentManager.assign(daughter, true);
Please file a JIRA.
Thanks for reporting this issue.
On Mon, Jul 4, 2011 at 7:10 PM, Weihua JIANG <we...@gmail.com> wrote:
> According to RS log, it seems it performs a region split during this
> RS shutdown process. Thus, RS only performs the split and add the two
> daughter regions to META, but not open them (MetaEditor.addDaughter()
> with serverInfo as null). It seems it's the correct behavior for RS.
>
> But, I can't find the corresponding code on master side to assign
> these unassigned regions. And the log on master side doesn't show an
> hint of such assignment decision.
>
> Thanks
> Weihua
>
> 2011/7/5 Weihua JIANG <we...@gmail.com>:
> > Both daughter regions are offline.
> >
> > The full log of master related to this RS down are:
> >
> > 2011-06-30 22:56:51,621 INFO
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
> > ephemeral node deleted, processing expiration
> > [xmlqa-clv4.sh.intel.com,50820,1309421874765]
> > 2011-06-30 22:56:51,622 DEBUG
> > org.apache.hadoop.hbase.master.ServerManager:
> > Added=xmlqa-clv4.sh.intel.com,50820,1309421874765 to dead servers,
> > submitted shutdown handler to be executed, root=false, meta=false
> > 2011-06-30 22:56:51,622 INFO
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> > Splitting logs for xmlqa-clv4.sh.intel.com,50820,1309421874765
> > 2011-06-30 22:56:51,633 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 35
> > hlog(s) in hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> > 2011-06-30 22:56:51,633 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> > Thread[WriterThread-0,5,main]: starting
> > 2011-06-30 22:56:51,633 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> > Thread[WriterThread-1,5,main]: starting
> > 2011-06-30 22:56:51,633 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 1 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> ,
> > length=63870691
> > 2011-06-30 22:56:51,634 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> > 2011-06-30 22:56:51,634 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> > Thread[WriterThread-2,5,main]: starting
> > 2011-06-30 22:56:52,636 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> > 2011-06-30 22:56:52,736 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,736 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> > region=ed60ec735e30db1d99290995eb1cd2d7
> > 2011-06-30 22:56:52,777 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,777 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> > region=625da13f7c9f8d2a6db3e7bd892de061
> > 2011-06-30 22:56:52,790 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,790 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> > region=6a22ac864ca20a7f34b8822b6b092697
> > 2011-06-30 22:56:52,825 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,826 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> > region=0ea99d0d995b95e8ea07441d30aa5545
> > 2011-06-30 22:56:52,836 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,836 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> > region=32c8d35a95c6ac4f399c17dfa04066b1
> > 2011-06-30 22:56:52,841 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,841 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> > region=d5a18e344d77bdccb568fd59623fb566
> > 2011-06-30 22:56:52,868 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,868 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> > region=e4a2320bc8ff30fee1a9b20267b399cf
> > 2011-06-30 22:56:52,909 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:52,909 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> > region=de5cb72653d016804cbd16f4a71470cd
> > 2011-06-30 22:56:53,017 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,018 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> > region=5a4a923e81c6127eb7f3ed61e5774063
> > 2011-06-30 22:56:53,023 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,024 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> > region=3e0c81217d4b47920e11960f1816efae
> > 2011-06-30 22:56:53,069 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,069 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> > region=d673806c5348d8517dcdb3392e5f5b21
> > 2011-06-30 22:56:53,101 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,101 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> > region=7cfa5c93f2d46d01c8f60ff934053ad1
> > 2011-06-30 22:56:53,109 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,109 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> > region=84c6f6ce13ffbeb3eb83c765fef754f3
> > 2011-06-30 22:56:53,179 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,179 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> > region=c94dc395183fd5a2286ec823b194b517
> > 2011-06-30 22:56:53,190 INFO
> > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> > syncFs -- HDFS-200
> > 2011-06-30 22:56:53,190 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> > path=hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> > region=0dd9c11896eca61505861a9e49f2fef3
> > 2011-06-30 22:56:54,198 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25752
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> > 2011-06-30 22:56:54,198 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 2 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> ,
> > length=73429706
> > 2011-06-30 22:56:54,198 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> > 2011-06-30 22:56:55,201 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> > 2011-06-30 22:56:57,787 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=29756
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> > 2011-06-30 22:56:57,787 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 3 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> ,
> > length=65063774
> > 2011-06-30 22:56:57,788 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> > 2011-06-30 22:56:58,793 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> > 2011-06-30 22:57:00,533 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26415
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> > 2011-06-30 22:57:00,533 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 4 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> ,
> > length=64631565
> > 2011-06-30 22:57:00,533 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> > 2011-06-30 22:57:01,537 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> > 2011-06-30 22:57:03,755 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25941
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> > 2011-06-30 22:57:03,755 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 5 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> ,
> > length=68367834
> > 2011-06-30 22:57:03,755 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> > 2011-06-30 22:57:04,757 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> > 2011-06-30 22:57:06,463 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27679
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> > 2011-06-30 22:57:06,463 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 6 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> ,
> > length=64503295
> > 2011-06-30 22:57:06,463 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> > 2011-06-30 22:57:07,465 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> > 2011-06-30 22:57:09,089 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26171
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> > 2011-06-30 22:57:09,089 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 7 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> ,
> > length=64201147
> > 2011-06-30 22:57:09,089 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> > 2011-06-30 22:57:10,093 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> > 2011-06-30 22:57:11,792 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25904
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> > 2011-06-30 22:57:11,792 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 8 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> ,
> > length=63800579
> > 2011-06-30 22:57:11,792 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> > 2011-06-30 22:57:12,794 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> > 2011-06-30 22:57:18,504 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> > 2011-06-30 22:57:18,504 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 9 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> ,
> > length=64119848
> > 2011-06-30 22:57:18,504 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> > 2011-06-30 22:57:19,506 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> > 2011-06-30 22:57:21,160 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26023
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> > 2011-06-30 22:57:21,160 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 10 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> ,
> > length=64643463
> > 2011-06-30 22:57:21,160 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> > 2011-06-30 22:57:22,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> > 2011-06-30 22:57:23,541 DEBUG
> > org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> > currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> > dead server
> > 2011-06-30 22:57:23,573 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> > Server listener on 50800: readAndProcess threw exception
> > java.io.IOException: Connection reset by peer. Count of bytes read: 0
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:206)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1357)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > 2011-06-30 22:57:23,674 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26324
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> > 2011-06-30 22:57:23,674 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 11 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> ,
> > length=64232645
> > 2011-06-30 22:57:23,674 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> > 2011-06-30 22:57:23,734 DEBUG
> > org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> > currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> > dead server
> > 2011-06-30 22:57:24,678 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> > 2011-06-30 22:57:26,287 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25901
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> > 2011-06-30 22:57:26,288 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 12 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> ,
> > length=68916014
> > 2011-06-30 22:57:26,288 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> > 2011-06-30 22:57:27,291 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> > 2011-06-30 22:57:29,540 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27997
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> > 2011-06-30 22:57:29,540 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 13 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> ,
> > length=64023210
> > 2011-06-30 22:57:29,541 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> > 2011-06-30 22:57:30,543 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> > 2011-06-30 22:57:32,118 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25852
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> > 2011-06-30 22:57:32,118 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 14 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> ,
> > length=63963259
> > 2011-06-30 22:57:32,118 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> > 2011-06-30 22:57:33,123 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> > 2011-06-30 22:57:36,027 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25800
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> > 2011-06-30 22:57:36,027 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 15 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> ,
> > length=65390175
> > 2011-06-30 22:57:36,028 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> > 2011-06-30 22:57:37,031 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> > 2011-06-30 22:57:38,618 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26591
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> > 2011-06-30 22:57:38,618 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 16 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> ,
> > length=64082403
> > 2011-06-30 22:57:38,618 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> > 2011-06-30 22:57:39,623 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> > 2011-06-30 22:57:42,129 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26079
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> > 2011-06-30 22:57:42,129 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 17 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> ,
> > length=64438939
> > 2011-06-30 22:57:42,129 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> > 2011-06-30 22:57:43,132 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> > 2011-06-30 22:57:44,965 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26012
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> > 2011-06-30 22:57:44,965 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 18 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> ,
> > length=64295447
> > 2011-06-30 22:57:44,965 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> > 2011-06-30 22:57:45,968 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> > 2011-06-30 22:57:48,826 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26038
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> > 2011-06-30 22:57:48,826 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 19 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> ,
> > length=64592159
> > 2011-06-30 22:57:48,826 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> > 2011-06-30 22:57:49,828 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> > 2011-06-30 22:57:53,219 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26061
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> > 2011-06-30 22:57:53,219 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 20 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> ,
> > length=64057243
> > 2011-06-30 22:57:53,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> > 2011-06-30 22:57:54,224 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> > 2011-06-30 22:57:58,220 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25765
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> > 2011-06-30 22:57:58,220 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 21 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> ,
> > length=64306648
> > 2011-06-30 22:57:58,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> > 2011-06-30 22:57:59,225 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> > 2011-06-30 22:58:00,572 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26006
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> > 2011-06-30 22:58:00,572 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 22 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> ,
> > length=64238208
> > 2011-06-30 22:58:00,572 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> > 2011-06-30 22:58:01,577 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> > 2011-06-30 22:58:04,010 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25961
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> > 2011-06-30 22:58:04,010 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 23 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> ,
> > length=66022861
> > 2011-06-30 22:58:04,011 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> > 2011-06-30 22:58:05,017 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> > 2011-06-30 22:58:08,583 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26599
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> > 2011-06-30 22:58:08,583 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 24 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> ,
> > length=64341194
> > 2011-06-30 22:58:08,583 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> > 2011-06-30 22:58:09,585 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> > 2011-06-30 22:58:11,171 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25978
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> > 2011-06-30 22:58:11,171 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 25 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> ,
> > length=64083581
> > 2011-06-30 22:58:11,171 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> > 2011-06-30 22:58:12,173 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> > 2011-06-30 22:58:14,655 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26044
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> > 2011-06-30 22:58:14,656 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 26 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> ,
> > length=64093785
> > 2011-06-30 22:58:14,656 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> > 2011-06-30 22:58:15,658 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> > 2011-06-30 22:58:17,914 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25712
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> > 2011-06-30 22:58:17,915 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 27 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> ,
> > length=64115777
> > 2011-06-30 22:58:17,915 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> > 2011-06-30 22:58:18,918 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> > 2011-06-30 22:58:20,762 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> > 2011-06-30 22:58:20,762 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 28 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> ,
> > length=64628500
> > 2011-06-30 22:58:20,762 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> > 2011-06-30 22:58:21,766 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> > 2011-06-30 22:58:23,158 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26131
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> > 2011-06-30 22:58:23,159 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 29 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> ,
> > length=64522869
> > 2011-06-30 22:58:23,159 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> > 2011-06-30 22:58:24,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> > 2011-06-30 22:58:25,593 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26159
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> > 2011-06-30 22:58:25,593 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 30 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> ,
> > length=64127204
> > 2011-06-30 22:58:25,593 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> > 2011-06-30 22:58:26,594 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> > 2011-06-30 22:58:28,023 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> > 2011-06-30 22:58:28,023 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 31 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> ,
> > length=96405599
> > 2011-06-30 22:58:28,023 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> > 2011-06-30 22:58:29,026 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> > 2011-06-30 22:58:31,603 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38922
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> > 2011-06-30 22:58:31,603 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 32 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> ,
> > length=64219650
> > 2011-06-30 22:58:31,603 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> > 2011-06-30 22:58:32,607 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> > 2011-06-30 22:58:33,924 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> > 2011-06-30 22:58:33,924 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 33 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> ,
> > length=64734596
> > 2011-06-30 22:58:33,924 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> > 2011-06-30 22:58:34,927 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> > 2011-06-30 22:58:37,455 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26330
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> > 2011-06-30 22:58:37,456 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 34 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> ,
> > length=95982267
> > 2011-06-30 22:58:37,456 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> > 2011-06-30 22:58:38,459 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> > 2011-06-30 22:58:40,216 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38754
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> > 2011-06-30 22:58:40,216 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> > 35 of 35: hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> ,
> > length=0
> > 2011-06-30 22:58:40,216 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Recovering file
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > 2011-06-30 22:58:41,219 INFO org.apache.hadoop.hbase.util.FSUtils:
> > Finished lease recover attempt for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > 2011-06-30 22:58:41,219 WARN
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > might be still open, length is 0
> > 2011-06-30 22:58:42,146 DEBUG
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=23462
> > entries from hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > 2011-06-30 22:58:42,179 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> > 2011-06-30 22:58:42,185 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> > 2011-06-30 22:58:42,191 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> > 2011-06-30 22:58:42,196 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> > 2011-06-30 22:58:42,203 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> > 2011-06-30 22:58:42,255 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> > 2011-06-30 22:58:42,270 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> > 2011-06-30 22:58:42,276 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> > 2011-06-30 22:58:42,292 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> > 2011-06-30 22:58:42,300 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> > 2011-06-30 22:58:42,306 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> > 2011-06-30 22:58:42,321 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> > 2011-06-30 22:58:42,329 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> > 2011-06-30 22:58:42,336 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> > 2011-06-30 22:58:42,342 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> > 2011-06-30 22:58:42,348 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> > 2011-06-30 22:58:42,354 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> > 2011-06-30 22:58:42,360 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> > 2011-06-30 22:58:42,366 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> > 2011-06-30 22:58:42,372 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> > 2011-06-30 22:58:42,378 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> > 2011-06-30 22:58:42,408 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> > 2011-06-30 22:58:42,414 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> > 2011-06-30 22:58:42,420 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> > 2011-06-30 22:58:42,426 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> > 2011-06-30 22:58:42,437 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> > 2011-06-30 22:58:42,444 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> > 2011-06-30 22:58:42,491 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> > 2011-06-30 22:58:42,504 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> > 2011-06-30 22:58:42,510 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> > 2011-06-30 22:58:42,516 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> > 2011-06-30 22:58:42,522 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> > 2011-06-30 22:58:42,528 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> > 2011-06-30 22:58:42,534 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> > 2011-06-30 22:58:42,540 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> > processed log hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > to hdfs://
> namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> > 2011-06-30 22:58:42,546 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for
> > split writer threads to finish
> > 2011-06-30 22:58:49,942 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers
> > finished
> > 2011-06-30 22:58:50,002 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> > (wrote 62511 edits in 6859ms)
> > 2011-06-30 22:58:50,014 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> > (wrote 63273 edits in 9130ms)
> > 2011-06-30 22:58:51,226 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> > (wrote 63193 edits in 29376ms)
> > 2011-06-30 22:58:51,232 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> > (wrote 62727 edits in 16595ms)
> > 2011-06-30 22:58:52,413 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> > (wrote 62672 edits in 7177ms)
> > 2011-06-30 22:58:52,461 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> > (wrote 62636 edits in 52110ms)
> > 2011-06-30 22:58:52,509 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> > (wrote 62860 edits in 19402ms)
> > 2011-06-30 22:58:52,515 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> > (wrote 62718 edits in 12592ms)
> > 2011-06-30 22:58:52,605 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> > (wrote 62561 edits in 12576ms)
> > 2011-06-30 22:58:52,670 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> > (wrote 62860 edits in 9056ms)
> > 2011-06-30 22:58:52,719 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> > (wrote 63328 edits in 8381ms)
> > 2011-06-30 22:58:52,725 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> > (wrote 62040 edits in 7900ms)
> > 2011-06-30 22:58:52,731 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> > (wrote 62397 edits in 19988ms)
> > 2011-06-30 22:58:52,779 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> > (wrote 62952 edits in 21391ms)
> > 2011-06-30 22:58:52,785 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> > hdfs://
> namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> > (wrote 62765 edits in 15781ms)
> > 2011-06-30 22:58:52,785 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
> > splitting completed in 121163 ms for
> > hdfs://
> namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> > 2011-06-30 22:58:52,893 INFO
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> > Reassigning 16 region(s) that
> > xmlqa-clv4.sh.intel.com,50820,1309421874765 was carrying (skipping 0
> > regions(s) that are already in transition)
> > 2011-06-30 22:58:52,894 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 6a22ac864ca20a7f34b8822b6b092697 with OFFLINE state
> > 2011-06-30 22:58:52,896 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.,
> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:52,896 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> > to hadoop05.sh.intel.com,50820,1309421836726
> > 2011-06-30 22:58:52,898 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 625da13f7c9f8d2a6db3e7bd892de061 with OFFLINE state
> > 2011-06-30 22:58:52,900 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.,
> > src=, dest=harpertown083.sh.intel.com,50820,1309421853805; 10
> > (online=10, exclude=null) available servers
> > 2011-06-30 22:58:52,900 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> > to harpertown083.sh.intel.com,50820,1309421853805
> > 2011-06-30 22:58:52,900 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop05.sh.intel.com,50820,1309421836726,
> > region=6a22ac864ca20a7f34b8822b6b092697
> > 2011-06-30 22:58:52,902 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 32c8d35a95c6ac4f399c17dfa04066b1 with OFFLINE state
> > 2011-06-30 22:58:52,904 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.,
> > src=, dest=xmlqa-clv6.sh.intel.com,50820,1309421880784; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:52,904 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> > to xmlqa-clv6.sh.intel.com,50820,1309421880784
> > 2011-06-30 22:58:52,904 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=harpertown083.sh.intel.com,50820,1309421853805,
> > region=625da13f7c9f8d2a6db3e7bd892de061
> > 2011-06-30 22:58:52,906 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 0ea99d0d995b95e8ea07441d30aa5545 with OFFLINE state
> > 2011-06-30 22:58:52,935 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.,
> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:52,935 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> > to hadoop05.sh.intel.com,50820,1309421836726
> > 2011-06-30 22:58:52,936 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=xmlqa-clv6.sh.intel.com,50820,1309421880784,
> > region=32c8d35a95c6ac4f399c17dfa04066b1
> > 2011-06-30 22:58:52,937 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for d5a18e344d77bdccb568fd59623fb566 with OFFLINE state
> > 2011-06-30 22:58:52,939 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.,
> > src=, dest=hbase01.sh.intel.com,50820,1309421885586; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:52,939 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> > to hbase01.sh.intel.com,50820,1309421885586
> > 2011-06-30 22:58:52,940 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop05.sh.intel.com,50820,1309421836726,
> > region=0ea99d0d995b95e8ea07441d30aa5545
> > 2011-06-30 22:58:52,941 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for ed60ec735e30db1d99290995eb1cd2d7 with OFFLINE state
> > 2011-06-30 22:58:52,943 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hbase01.sh.intel.com,50820,1309421885586,
> > region=d5a18e344d77bdccb568fd59623fb566
> > 2011-06-30 22:58:52,943 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.,
> > src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:52,943 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> > to hadoop01.sh.intel.com,50820,1309421825940
> > 2011-06-30 22:58:52,945 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> > and split region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> > checking daughter presence
> > 2011-06-30 22:58:52,946 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop01.sh.intel.com,50820,1309421825940,
> > region=ed60ec735e30db1d99290995eb1cd2d7
> > 2011-06-30 22:58:53,005 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > present
> > 2011-06-30 22:58:53,065 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > present
> > 2011-06-30 22:58:53,065 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 3e0c81217d4b47920e11960f1816efae with OFFLINE state
> > 2011-06-30 22:58:53,067 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.,
> > src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,067 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> > to hadoop01.sh.intel.com,50820,1309421825940
> > 2011-06-30 22:58:53,070 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for e4a2320bc8ff30fee1a9b20267b399cf with OFFLINE state
> > 2011-06-30 22:58:53,072 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop01.sh.intel.com,50820,1309421825940,
> > region=3e0c81217d4b47920e11960f1816efae
> > 2011-06-30 22:58:53,072 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.,
> > src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,072 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> > to hadoop06.sh.intel.com,50820,1309421846110
> > 2011-06-30 22:58:53,083 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 5a4a923e81c6127eb7f3ed61e5774063 with OFFLINE state
> > 2011-06-30 22:58:53,085 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.,
> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,085 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> > to ditest02.sh.intel.com,50820,1309421803646
> > 2011-06-30 22:58:53,085 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop06.sh.intel.com,50820,1309421846110,
> > region=e4a2320bc8ff30fee1a9b20267b399cf
> > 2011-06-30 22:58:53,086 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for d673806c5348d8517dcdb3392e5f5b21 with OFFLINE state
> > 2011-06-30 22:58:53,087 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.,
> > src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,087 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> > to hadoop06.sh.intel.com,50820,1309421846110
> > 2011-06-30 22:58:53,088 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=ditest02.sh.intel.com,50820,1309421803646,
> > region=5a4a923e81c6127eb7f3ed61e5774063
> > 2011-06-30 22:58:53,090 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 7cfa5c93f2d46d01c8f60ff934053ad1 with OFFLINE state
> > 2011-06-30 22:58:53,117 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.,
> > src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,117 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> > to hadoop05.sh.intel.com,50820,1309421836726
> > 2011-06-30 22:58:53,117 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop06.sh.intel.com,50820,1309421846110,
> > region=d673806c5348d8517dcdb3392e5f5b21
> > 2011-06-30 22:58:53,120 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 84c6f6ce13ffbeb3eb83c765fef754f3 with OFFLINE state
> > 2011-06-30 22:58:53,126 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.,
> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,126 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> > to ditest02.sh.intel.com,50820,1309421803646
> > 2011-06-30 22:58:53,127 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop05.sh.intel.com,50820,1309421836726,
> > region=7cfa5c93f2d46d01c8f60ff934053ad1
> > 2011-06-30 22:58:53,129 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for c94dc395183fd5a2286ec823b194b517 with OFFLINE state
> > 2011-06-30 22:58:53,143 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.,
> > src=, dest=ditest01.sh.intel.com,50820,1309421799506; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,143 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> > to ditest01.sh.intel.com,50820,1309421799506
> > 2011-06-30 22:58:53,143 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=ditest02.sh.intel.com,50820,1309421803646,
> > region=84c6f6ce13ffbeb3eb83c765fef754f3
> > 2011-06-30 22:58:53,149 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=ditest01.sh.intel.com,50820,1309421799506,
> > region=c94dc395183fd5a2286ec823b194b517
> > 2011-06-30 22:58:53,151 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for 0dd9c11896eca61505861a9e49f2fef3 with OFFLINE state
> > 2011-06-30 22:58:53,153 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.,
> > src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> > exclude=null) available servers
> > 2011-06-30 22:58:53,153 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> > to ditest02.sh.intel.com,50820,1309421803646
> > 2011-06-30 22:58:53,156 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> > for d9adac25d8b83a3d4c252acaf5ba17af with OFFLINE state
> > 2011-06-30 22:58:53,157 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition plan was found (or we are ignoring an existing plan) for
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> > so generated a random one;
> >
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.,
> > src=, dest=harpertown0816.sh.intel.com,50820,1309421864349; 10
> > (online=10, exclude=null) available servers
> > 2011-06-30 22:58:53,157 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> > to harpertown0816.sh.intel.com,50820,1309421864349
> > 2011-06-30 22:58:53,158 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=ditest02.sh.intel.com,50820,1309421803646,
> > region=0dd9c11896eca61505861a9e49f2fef3
> > 2011-06-30 22:58:53,159 INFO
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
> > processing of shutdown of xmlqa-clv4.sh.intel.com,50820,1309421874765
> >
> >
> > Thanks
> > Weihua
> >
> > 2011/7/4 Ted Yu <yu...@gmail.com>:
> >> In the future, please direct questions on cdh releases to
> >> cdh-dev@cloudera.org
> >> You may cc dev@hbase.apache.org
> >>
> >> There is more than one minute difference between master and RS logs.
> >> Which one of the daughter regions didn't come online ?
> >>
> >> Cheers
> >>
> >> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com>
> wrote:
> >>
> >>> The HBase version we are using is CDH3U0.
> >>>
> >>> Thanks
> >>> Weihua
> >>>
> >>> 2011/7/4 Weihua JIANG <we...@gmail.com>:
> >>> > Hi all,
> >>> >
> >>> > We encountered a problem about region not onlining. A region is
> >>> > splitted by a closing RS and then this RS down. It seems master has
> >>> > known this split but it doesn't tried to make it online. Log from
> >>> > master
> >>> > 2011-06-30 22:58:52,945 DEBUG
> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Offlined
> >>> > and split region
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> >>> > checking daughter presence
> >>> > 2011-06-30 22:58:52,946 DEBUG
> >>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> >>> > transition=RS_ZK_REGION_OPENING,
> >>> > server=hadoop01.sh.intel.com,50820,1309421825940,
> >>> > region=ed60ec735e30db1d99290995eb1cd2d7
> >>> > 2011-06-30 22:58:53,005 DEBUG
> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Daughter
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> >>> > present
> >>> > 2011-06-30 22:58:53,065 DEBUG
> >>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Daughter
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> >>> > present
> >>> >
> >>> > Log from RS is:
> >>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> >>> > Server handler 73 on 50820 caught:
> >>> > java.nio.channels.ClosedChannelException
> >>> > at
> >>>
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> >>> > at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >>> > at
> >>>
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
> >>> > at
> >>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> >>> > at
> >>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> >>> > at
> >>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
> >>> >
> >>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> >>> > Server handler 73 on 50820: exiting
> >>> > 2011-06-30 22:57:05,767 INFO
> >>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820
> closing
> >>> > leases
> >>> > 2011-06-30 22:57:05,768 INFO
> >>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
> >>> > leases
> >>> > 2011-06-30 22:57:05,768 INFO
> >>> >
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >>> > Closed zookeeper sessionid=0x130ba69074900b4
> >>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> >>> > 0x130ba69074900b4 closed
> >>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
> >>> > EventThread shut down
> >>> > 2011-06-30 22:57:05,857 DEBUG
> >>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> >>> > 2011-06-30 22:57:05,863 DEBUG
> >>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> >>> > 2011-06-30 22:57:05,911 INFO
> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
> >>> > in META
> >>> > 2011-06-30 22:57:05,942 INFO
> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> >>> > in region .META.,,1, serverInfo=null
> >>> > 2011-06-30 22:57:05,943 INFO
> >>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> >>> > daughter
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> >>> > because stopping=false, stopped=true
> >>> > 2011-06-30 22:57:05,950 INFO
> >>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> >>> > in region .META.,,1, serverInfo=null
> >>> > 2011-06-30 22:57:05,950 INFO
> >>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> >>> > daughter
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> >>> > because stopping=false, stopped=true
> >>> > 2011-06-30 22:57:06,004 INFO
> >>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> >>> > updated, and report to master.
> >>> >
> >>>
> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
> >>> > new regions:
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
> >>> >
> >>>
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
> >>> > Split took 1mins, 12sec
> >>> > 2011-06-30 22:57:06,004 DEBUG
> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> >>> > Split Thread to finish...
> >>> > 2011-06-30 22:57:06,004 DEBUG
> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> >>> > Large Compaction Thread to finish...
> >>> > 2011-06-30 22:57:06,004 DEBUG
> >>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> >>> > Small Compaction Thread to finish...
> >>> > 2011-06-30 22:57:06,004 INFO
> >>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
> >>> > exiting
> >>> > 2011-06-30 22:57:06,090 INFO
> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >>> > starting; hbase.shutdown.hook=true;
> >>> > fsShutdownHook=Thread[Thread-15,5,main]
> >>> > 2011-06-30 22:57:06,090 INFO
> >>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> >>> > hook
> >>> > 2011-06-30 22:57:06,090 INFO
> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> >>> > shutdown hook thread.
> >>> > 2011-06-30 22:57:06,196 INFO
> >>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> >>> > finished.
> >>> >
> >>> >
> >>> > Thanks
> >>> > Weihua
> >>> >
> >>>
> >>
> >
>
Re: Region not online after split by a closing RS
Posted by Weihua JIANG <we...@gmail.com>.
According to RS log, it seems it performs a region split during this
RS shutdown process. Thus, RS only performs the split and add the two
daughter regions to META, but not open them (MetaEditor.addDaughter()
with serverInfo as null). It seems it's the correct behavior for RS.
But, I can't find the corresponding code on master side to assign
these unassigned regions. And the log on master side doesn't show an
hint of such assignment decision.
Thanks
Weihua
2011/7/5 Weihua JIANG <we...@gmail.com>:
> Both daughter regions are offline.
>
> The full log of master related to this RS down are:
>
> 2011-06-30 22:56:51,621 INFO
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
> ephemeral node deleted, processing expiration
> [xmlqa-clv4.sh.intel.com,50820,1309421874765]
> 2011-06-30 22:56:51,622 DEBUG
> org.apache.hadoop.hbase.master.ServerManager:
> Added=xmlqa-clv4.sh.intel.com,50820,1309421874765 to dead servers,
> submitted shutdown handler to be executed, root=false, meta=false
> 2011-06-30 22:56:51,622 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Splitting logs for xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:56:51,633 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 35
> hlog(s) in hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-0,5,main]: starting
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-1,5,main]: starting
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 1 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944,
> length=63870691
> 2011-06-30 22:56:51,634 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:51,634 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-2,5,main]: starting
> 2011-06-30 22:56:52,636 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:52,736 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,736 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:56:52,777 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,777 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> region=625da13f7c9f8d2a6db3e7bd892de061
> 2011-06-30 22:56:52,790 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,790 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> region=6a22ac864ca20a7f34b8822b6b092697
> 2011-06-30 22:56:52,825 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> region=0ea99d0d995b95e8ea07441d30aa5545
> 2011-06-30 22:56:52,836 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,836 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> region=32c8d35a95c6ac4f399c17dfa04066b1
> 2011-06-30 22:56:52,841 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,841 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> region=d5a18e344d77bdccb568fd59623fb566
> 2011-06-30 22:56:52,868 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,868 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> region=e4a2320bc8ff30fee1a9b20267b399cf
> 2011-06-30 22:56:52,909 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,909 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> region=de5cb72653d016804cbd16f4a71470cd
> 2011-06-30 22:56:53,017 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,018 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> region=5a4a923e81c6127eb7f3ed61e5774063
> 2011-06-30 22:56:53,023 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,024 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> region=3e0c81217d4b47920e11960f1816efae
> 2011-06-30 22:56:53,069 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,069 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> region=d673806c5348d8517dcdb3392e5f5b21
> 2011-06-30 22:56:53,101 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,101 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> region=7cfa5c93f2d46d01c8f60ff934053ad1
> 2011-06-30 22:56:53,109 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,109 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> region=84c6f6ce13ffbeb3eb83c765fef754f3
> 2011-06-30 22:56:53,179 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,179 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> region=c94dc395183fd5a2286ec823b194b517
> 2011-06-30 22:56:53,190 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,190 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> region=0dd9c11896eca61505861a9e49f2fef3
> 2011-06-30 22:56:54,198 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25752
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:54,198 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 2 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636,
> length=73429706
> 2011-06-30 22:56:54,198 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:55,201 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:57,787 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=29756
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:57,787 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 3 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577,
> length=65063774
> 2011-06-30 22:56:57,788 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:56:58,793 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:57:00,533 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26415
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:57:00,533 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 4 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792,
> length=64631565
> 2011-06-30 22:57:00,533 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:01,537 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:03,755 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25941
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:03,755 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 5 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002,
> length=68367834
> 2011-06-30 22:57:03,755 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:04,757 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:06,463 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27679
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:06,463 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 6 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699,
> length=64503295
> 2011-06-30 22:57:06,463 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:07,465 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:09,089 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26171
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:09,089 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 7 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029,
> length=64201147
> 2011-06-30 22:57:09,089 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:10,093 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:11,792 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25904
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:11,792 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 8 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992,
> length=63800579
> 2011-06-30 22:57:11,792 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:12,794 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:18,504 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:18,504 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 9 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571,
> length=64119848
> 2011-06-30 22:57:18,504 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:19,506 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:21,160 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26023
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:21,160 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 10 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271,
> length=64643463
> 2011-06-30 22:57:21,160 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:22,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:23,541 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> dead server
> 2011-06-30 22:57:23,573 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server listener on 50800: readAndProcess threw exception
> java.io.IOException: Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:206)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1357)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2011-06-30 22:57:23,674 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26324
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:23,674 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 11 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420,
> length=64232645
> 2011-06-30 22:57:23,674 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:23,734 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> dead server
> 2011-06-30 22:57:24,678 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:26,287 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25901
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:26,288 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 12 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336,
> length=68916014
> 2011-06-30 22:57:26,288 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:27,291 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:29,540 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27997
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:29,540 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 13 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618,
> length=64023210
> 2011-06-30 22:57:29,541 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:30,543 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:32,118 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25852
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:32,118 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 14 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430,
> length=63963259
> 2011-06-30 22:57:32,118 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:33,123 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:36,027 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25800
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:36,027 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 15 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319,
> length=65390175
> 2011-06-30 22:57:36,028 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:37,031 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:38,618 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26591
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:38,618 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 16 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600,
> length=64082403
> 2011-06-30 22:57:38,618 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:39,623 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:42,129 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26079
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:42,129 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 17 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088,
> length=64438939
> 2011-06-30 22:57:42,129 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:43,132 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:44,965 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26012
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:44,965 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 18 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332,
> length=64295447
> 2011-06-30 22:57:44,965 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:45,968 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:48,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26038
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:48,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 19 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952,
> length=64592159
> 2011-06-30 22:57:48,826 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:49,828 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:53,219 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26061
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:53,219 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 20 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863,
> length=64057243
> 2011-06-30 22:57:53,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:54,224 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:58,220 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25765
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:58,220 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 21 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965,
> length=64306648
> 2011-06-30 22:57:58,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:57:59,225 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:00,572 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26006
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:00,572 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 22 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999,
> length=64238208
> 2011-06-30 22:58:00,572 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:01,577 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:04,010 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25961
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:04,010 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 23 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242,
> length=66022861
> 2011-06-30 22:58:04,011 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:05,017 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:08,583 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26599
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:08,583 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 24 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462,
> length=64341194
> 2011-06-30 22:58:08,583 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:09,585 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:11,171 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25978
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:11,171 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 25 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064,
> length=64083581
> 2011-06-30 22:58:11,171 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:12,173 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:14,655 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26044
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:14,656 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 26 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989,
> length=64093785
> 2011-06-30 22:58:14,656 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:15,658 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:17,914 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25712
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:17,915 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 27 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493,
> length=64115777
> 2011-06-30 22:58:17,915 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:18,918 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:20,762 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:20,762 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 28 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468,
> length=64628500
> 2011-06-30 22:58:20,762 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:21,766 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:23,158 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26131
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:23,159 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 29 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048,
> length=64522869
> 2011-06-30 22:58:23,159 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:24,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:25,593 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26159
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:25,593 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 30 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226,
> length=64127204
> 2011-06-30 22:58:25,593 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:26,594 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:28,023 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:28,023 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 31 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208,
> length=96405599
> 2011-06-30 22:58:28,023 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:29,026 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:31,603 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38922
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:31,603 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 32 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723,
> length=64219650
> 2011-06-30 22:58:31,603 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:32,607 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:33,924 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:33,924 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 33 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548,
> length=64734596
> 2011-06-30 22:58:33,924 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:34,927 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:37,455 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26330
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:37,456 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 34 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138,
> length=95982267
> 2011-06-30 22:58:37,456 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:38,459 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:40,216 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38754
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:40,216 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 35 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661,
> length=0
> 2011-06-30 22:58:40,216 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:41,219 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:41,219 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> might be still open, length is 0
> 2011-06-30 22:58:42,146 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=23462
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:42,179 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:58:42,185 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:58:42,191 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:58:42,196 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:58:42,203 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:58:42,255 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:58:42,270 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:58:42,276 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:58:42,292 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:58:42,300 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:58:42,306 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:58:42,321 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:58:42,329 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:58:42,336 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:58:42,342 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:58:42,348 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:58:42,354 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:58:42,360 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:58:42,366 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:58:42,372 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:58:42,378 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:42,408 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:42,414 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:42,420 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:42,426 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:42,437 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:42,444 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:42,491 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:42,504 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:42,510 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:42,516 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:42,522 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:42,528 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:42,534 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:42,540 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:42,546 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for
> split writer threads to finish
> 2011-06-30 22:58:49,942 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers
> finished
> 2011-06-30 22:58:50,002 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> (wrote 62511 edits in 6859ms)
> 2011-06-30 22:58:50,014 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> (wrote 63273 edits in 9130ms)
> 2011-06-30 22:58:51,226 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> (wrote 63193 edits in 29376ms)
> 2011-06-30 22:58:51,232 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> (wrote 62727 edits in 16595ms)
> 2011-06-30 22:58:52,413 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> (wrote 62672 edits in 7177ms)
> 2011-06-30 22:58:52,461 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> (wrote 62636 edits in 52110ms)
> 2011-06-30 22:58:52,509 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> (wrote 62860 edits in 19402ms)
> 2011-06-30 22:58:52,515 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> (wrote 62718 edits in 12592ms)
> 2011-06-30 22:58:52,605 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> (wrote 62561 edits in 12576ms)
> 2011-06-30 22:58:52,670 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> (wrote 62860 edits in 9056ms)
> 2011-06-30 22:58:52,719 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> (wrote 63328 edits in 8381ms)
> 2011-06-30 22:58:52,725 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> (wrote 62040 edits in 7900ms)
> 2011-06-30 22:58:52,731 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> (wrote 62397 edits in 19988ms)
> 2011-06-30 22:58:52,779 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> (wrote 62952 edits in 21391ms)
> 2011-06-30 22:58:52,785 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> (wrote 62765 edits in 15781ms)
> 2011-06-30 22:58:52,785 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
> splitting completed in 121163 ms for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:58:52,893 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Reassigning 16 region(s) that
> xmlqa-clv4.sh.intel.com,50820,1309421874765 was carrying (skipping 0
> regions(s) that are already in transition)
> 2011-06-30 22:58:52,894 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 6a22ac864ca20a7f34b8822b6b092697 with OFFLINE state
> 2011-06-30 22:58:52,896 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,896 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:52,898 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 625da13f7c9f8d2a6db3e7bd892de061 with OFFLINE state
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.,
> src=, dest=harpertown083.sh.intel.com,50820,1309421853805; 10
> (online=10, exclude=null) available servers
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> to harpertown083.sh.intel.com,50820,1309421853805
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=6a22ac864ca20a7f34b8822b6b092697
> 2011-06-30 22:58:52,902 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 32c8d35a95c6ac4f399c17dfa04066b1 with OFFLINE state
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.,
> src=, dest=xmlqa-clv6.sh.intel.com,50820,1309421880784; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> to xmlqa-clv6.sh.intel.com,50820,1309421880784
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=harpertown083.sh.intel.com,50820,1309421853805,
> region=625da13f7c9f8d2a6db3e7bd892de061
> 2011-06-30 22:58:52,906 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 0ea99d0d995b95e8ea07441d30aa5545 with OFFLINE state
> 2011-06-30 22:58:52,935 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,935 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:52,936 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=xmlqa-clv6.sh.intel.com,50820,1309421880784,
> region=32c8d35a95c6ac4f399c17dfa04066b1
> 2011-06-30 22:58:52,937 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d5a18e344d77bdccb568fd59623fb566 with OFFLINE state
> 2011-06-30 22:58:52,939 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.,
> src=, dest=hbase01.sh.intel.com,50820,1309421885586; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,939 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> to hbase01.sh.intel.com,50820,1309421885586
> 2011-06-30 22:58:52,940 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=0ea99d0d995b95e8ea07441d30aa5545
> 2011-06-30 22:58:52,941 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for ed60ec735e30db1d99290995eb1cd2d7 with OFFLINE state
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hbase01.sh.intel.com,50820,1309421885586,
> region=d5a18e344d77bdccb568fd59623fb566
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.,
> src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> to hadoop01.sh.intel.com,50820,1309421825940
> 2011-06-30 22:58:52,945 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> and split region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> checking daughter presence
> 2011-06-30 22:58:52,946 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:58:53,005 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 3e0c81217d4b47920e11960f1816efae with OFFLINE state
> 2011-06-30 22:58:53,067 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.,
> src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,067 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> to hadoop01.sh.intel.com,50820,1309421825940
> 2011-06-30 22:58:53,070 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for e4a2320bc8ff30fee1a9b20267b399cf with OFFLINE state
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=3e0c81217d4b47920e11960f1816efae
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.,
> src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> to hadoop06.sh.intel.com,50820,1309421846110
> 2011-06-30 22:58:53,083 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 5a4a923e81c6127eb7f3ed61e5774063 with OFFLINE state
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop06.sh.intel.com,50820,1309421846110,
> region=e4a2320bc8ff30fee1a9b20267b399cf
> 2011-06-30 22:58:53,086 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d673806c5348d8517dcdb3392e5f5b21 with OFFLINE state
> 2011-06-30 22:58:53,087 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.,
> src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,087 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> to hadoop06.sh.intel.com,50820,1309421846110
> 2011-06-30 22:58:53,088 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=5a4a923e81c6127eb7f3ed61e5774063
> 2011-06-30 22:58:53,090 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 7cfa5c93f2d46d01c8f60ff934053ad1 with OFFLINE state
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop06.sh.intel.com,50820,1309421846110,
> region=d673806c5348d8517dcdb3392e5f5b21
> 2011-06-30 22:58:53,120 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 84c6f6ce13ffbeb3eb83c765fef754f3 with OFFLINE state
> 2011-06-30 22:58:53,126 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,126 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,127 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=7cfa5c93f2d46d01c8f60ff934053ad1
> 2011-06-30 22:58:53,129 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for c94dc395183fd5a2286ec823b194b517 with OFFLINE state
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.,
> src=, dest=ditest01.sh.intel.com,50820,1309421799506; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> to ditest01.sh.intel.com,50820,1309421799506
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=84c6f6ce13ffbeb3eb83c765fef754f3
> 2011-06-30 22:58:53,149 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest01.sh.intel.com,50820,1309421799506,
> region=c94dc395183fd5a2286ec823b194b517
> 2011-06-30 22:58:53,151 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 0dd9c11896eca61505861a9e49f2fef3 with OFFLINE state
> 2011-06-30 22:58:53,153 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,153 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,156 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d9adac25d8b83a3d4c252acaf5ba17af with OFFLINE state
> 2011-06-30 22:58:53,157 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.,
> src=, dest=harpertown0816.sh.intel.com,50820,1309421864349; 10
> (online=10, exclude=null) available servers
> 2011-06-30 22:58:53,157 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> to harpertown0816.sh.intel.com,50820,1309421864349
> 2011-06-30 22:58:53,158 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=0dd9c11896eca61505861a9e49f2fef3
> 2011-06-30 22:58:53,159 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
> processing of shutdown of xmlqa-clv4.sh.intel.com,50820,1309421874765
>
>
> Thanks
> Weihua
>
> 2011/7/4 Ted Yu <yu...@gmail.com>:
>> In the future, please direct questions on cdh releases to
>> cdh-dev@cloudera.org
>> You may cc dev@hbase.apache.org
>>
>> There is more than one minute difference between master and RS logs.
>> Which one of the daughter regions didn't come online ?
>>
>> Cheers
>>
>> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com> wrote:
>>
>>> The HBase version we are using is CDH3U0.
>>>
>>> Thanks
>>> Weihua
>>>
>>> 2011/7/4 Weihua JIANG <we...@gmail.com>:
>>> > Hi all,
>>> >
>>> > We encountered a problem about region not onlining. A region is
>>> > splitted by a closing RS and then this RS down. It seems master has
>>> > known this split but it doesn't tried to make it online. Log from
>>> > master
>>> > 2011-06-30 22:58:52,945 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>>> > and split region
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>>> > checking daughter presence
>>> > 2011-06-30 22:58:52,946 DEBUG
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>>> > transition=RS_ZK_REGION_OPENING,
>>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>>> > region=ed60ec735e30db1d99290995eb1cd2d7
>>> > 2011-06-30 22:58:53,005 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > present
>>> > 2011-06-30 22:58:53,065 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > present
>>> >
>>> > Log from RS is:
>>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>>> > Server handler 73 on 50820 caught:
>>> > java.nio.channels.ClosedChannelException
>>> > at
>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>>> > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>>> > at
>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>>> > at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>>> > at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>>> > at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>>> >
>>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> > Server handler 73 on 50820: exiting
>>> > 2011-06-30 22:57:05,767 INFO
>>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
>>> > leases
>>> > 2011-06-30 22:57:05,768 INFO
>>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>>> > leases
>>> > 2011-06-30 22:57:05,768 INFO
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>>> > Closed zookeeper sessionid=0x130ba69074900b4
>>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>>> > 0x130ba69074900b4 closed
>>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>>> > EventThread shut down
>>> > 2011-06-30 22:57:05,857 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > 2011-06-30 22:57:05,863 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > 2011-06-30 22:57:05,911 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>>> > in META
>>> > 2011-06-30 22:57:05,942 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > in region .META.,,1, serverInfo=null
>>> > 2011-06-30 22:57:05,943 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>>> > daughter
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > because stopping=false, stopped=true
>>> > 2011-06-30 22:57:05,950 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > in region .META.,,1, serverInfo=null
>>> > 2011-06-30 22:57:05,950 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>>> > daughter
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > because stopping=false, stopped=true
>>> > 2011-06-30 22:57:06,004 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>>> > updated, and report to master.
>>> >
>>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>>> > new regions:
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>>> > Split took 1mins, 12sec
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Split Thread to finish...
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Large Compaction Thread to finish...
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Small Compaction Thread to finish...
>>> > 2011-06-30 22:57:06,004 INFO
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>>> > exiting
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>> > starting; hbase.shutdown.hook=true;
>>> > fsShutdownHook=Thread[Thread-15,5,main]
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>>> > hook
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>>> > shutdown hook thread.
>>> > 2011-06-30 22:57:06,196 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>> > finished.
>>> >
>>> >
>>> > Thanks
>>> > Weihua
>>> >
>>>
>>
>
Re: Region not online after split by a closing RS
Posted by Weihua JIANG <we...@gmail.com>.
Both daughter regions are offline.
The full log of master related to this RS down are:
2011-06-30 22:56:51,621 INFO
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
ephemeral node deleted, processing expiration
[xmlqa-clv4.sh.intel.com,50820,1309421874765]
2011-06-30 22:56:51,622 DEBUG
org.apache.hadoop.hbase.master.ServerManager:
Added=xmlqa-clv4.sh.intel.com,50820,1309421874765 to dead servers,
submitted shutdown handler to be executed, root=false, meta=false
2011-06-30 22:56:51,622 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Splitting logs for xmlqa-clv4.sh.intel.com,50820,1309421874765
2011-06-30 22:56:51,633 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 35
hlog(s) in hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
2011-06-30 22:56:51,633 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
Thread[WriterThread-0,5,main]: starting
2011-06-30 22:56:51,633 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
Thread[WriterThread-1,5,main]: starting
2011-06-30 22:56:51,633 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
1 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944,
length=63870691
2011-06-30 22:56:51,634 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
2011-06-30 22:56:51,634 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
Thread[WriterThread-2,5,main]: starting
2011-06-30 22:56:52,636 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
2011-06-30 22:56:52,736 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,736 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
region=ed60ec735e30db1d99290995eb1cd2d7
2011-06-30 22:56:52,777 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,777 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
region=625da13f7c9f8d2a6db3e7bd892de061
2011-06-30 22:56:52,790 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,790 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
region=6a22ac864ca20a7f34b8822b6b092697
2011-06-30 22:56:52,825 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,826 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
region=0ea99d0d995b95e8ea07441d30aa5545
2011-06-30 22:56:52,836 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,836 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
region=32c8d35a95c6ac4f399c17dfa04066b1
2011-06-30 22:56:52,841 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,841 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
region=d5a18e344d77bdccb568fd59623fb566
2011-06-30 22:56:52,868 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,868 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
region=e4a2320bc8ff30fee1a9b20267b399cf
2011-06-30 22:56:52,909 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:52,909 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
region=de5cb72653d016804cbd16f4a71470cd
2011-06-30 22:56:53,017 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,018 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
region=5a4a923e81c6127eb7f3ed61e5774063
2011-06-30 22:56:53,023 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,024 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
region=3e0c81217d4b47920e11960f1816efae
2011-06-30 22:56:53,069 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,069 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
region=d673806c5348d8517dcdb3392e5f5b21
2011-06-30 22:56:53,101 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,101 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
region=7cfa5c93f2d46d01c8f60ff934053ad1
2011-06-30 22:56:53,109 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,109 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
region=84c6f6ce13ffbeb3eb83c765fef754f3
2011-06-30 22:56:53,179 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,179 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
region=c94dc395183fd5a2286ec823b194b517
2011-06-30 22:56:53,190 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
syncFs -- HDFS-200
2011-06-30 22:56:53,190 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
region=0dd9c11896eca61505861a9e49f2fef3
2011-06-30 22:56:54,198 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25752
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
2011-06-30 22:56:54,198 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
2 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636,
length=73429706
2011-06-30 22:56:54,198 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
2011-06-30 22:56:55,201 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
2011-06-30 22:56:57,787 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=29756
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
2011-06-30 22:56:57,787 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
3 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577,
length=65063774
2011-06-30 22:56:57,788 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
2011-06-30 22:56:58,793 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
2011-06-30 22:57:00,533 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26415
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
2011-06-30 22:57:00,533 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
4 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792,
length=64631565
2011-06-30 22:57:00,533 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
2011-06-30 22:57:01,537 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
2011-06-30 22:57:03,755 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25941
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
2011-06-30 22:57:03,755 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
5 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002,
length=68367834
2011-06-30 22:57:03,755 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
2011-06-30 22:57:04,757 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
2011-06-30 22:57:06,463 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27679
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
2011-06-30 22:57:06,463 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
6 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699,
length=64503295
2011-06-30 22:57:06,463 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
2011-06-30 22:57:07,465 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
2011-06-30 22:57:09,089 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26171
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
2011-06-30 22:57:09,089 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
7 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029,
length=64201147
2011-06-30 22:57:09,089 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
2011-06-30 22:57:10,093 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
2011-06-30 22:57:11,792 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25904
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
2011-06-30 22:57:11,792 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
8 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992,
length=63800579
2011-06-30 22:57:11,792 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
2011-06-30 22:57:12,794 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
2011-06-30 22:57:18,504 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
2011-06-30 22:57:18,504 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
9 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571,
length=64119848
2011-06-30 22:57:18,504 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
2011-06-30 22:57:19,506 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
2011-06-30 22:57:21,160 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26023
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
2011-06-30 22:57:21,160 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
10 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271,
length=64643463
2011-06-30 22:57:21,160 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
2011-06-30 22:57:22,162 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
2011-06-30 22:57:23,541 DEBUG
org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
dead server
2011-06-30 22:57:23,573 WARN org.apache.hadoop.ipc.HBaseServer: IPC
Server listener on 50800: readAndProcess threw exception
java.io.IOException: Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1357)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2011-06-30 22:57:23,674 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26324
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
2011-06-30 22:57:23,674 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
11 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420,
length=64232645
2011-06-30 22:57:23,674 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
2011-06-30 22:57:23,734 DEBUG
org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
dead server
2011-06-30 22:57:24,678 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
2011-06-30 22:57:26,287 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25901
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
2011-06-30 22:57:26,288 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
12 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336,
length=68916014
2011-06-30 22:57:26,288 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
2011-06-30 22:57:27,291 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
2011-06-30 22:57:29,540 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27997
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
2011-06-30 22:57:29,540 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
13 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618,
length=64023210
2011-06-30 22:57:29,541 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
2011-06-30 22:57:30,543 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
2011-06-30 22:57:32,118 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25852
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
2011-06-30 22:57:32,118 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
14 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430,
length=63963259
2011-06-30 22:57:32,118 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
2011-06-30 22:57:33,123 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
2011-06-30 22:57:36,027 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25800
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
2011-06-30 22:57:36,027 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
15 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319,
length=65390175
2011-06-30 22:57:36,028 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
2011-06-30 22:57:37,031 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
2011-06-30 22:57:38,618 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26591
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
2011-06-30 22:57:38,618 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
16 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600,
length=64082403
2011-06-30 22:57:38,618 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
2011-06-30 22:57:39,623 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
2011-06-30 22:57:42,129 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26079
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
2011-06-30 22:57:42,129 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
17 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088,
length=64438939
2011-06-30 22:57:42,129 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
2011-06-30 22:57:43,132 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
2011-06-30 22:57:44,965 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26012
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
2011-06-30 22:57:44,965 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
18 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332,
length=64295447
2011-06-30 22:57:44,965 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
2011-06-30 22:57:45,968 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
2011-06-30 22:57:48,826 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26038
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
2011-06-30 22:57:48,826 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
19 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952,
length=64592159
2011-06-30 22:57:48,826 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
2011-06-30 22:57:49,828 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
2011-06-30 22:57:53,219 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26061
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
2011-06-30 22:57:53,219 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
20 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863,
length=64057243
2011-06-30 22:57:53,220 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
2011-06-30 22:57:54,224 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
2011-06-30 22:57:58,220 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25765
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
2011-06-30 22:57:58,220 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
21 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965,
length=64306648
2011-06-30 22:57:58,220 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
2011-06-30 22:57:59,225 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
2011-06-30 22:58:00,572 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26006
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
2011-06-30 22:58:00,572 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
22 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999,
length=64238208
2011-06-30 22:58:00,572 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
2011-06-30 22:58:01,577 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
2011-06-30 22:58:04,010 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25961
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
2011-06-30 22:58:04,010 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
23 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242,
length=66022861
2011-06-30 22:58:04,011 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
2011-06-30 22:58:05,017 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
2011-06-30 22:58:08,583 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26599
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
2011-06-30 22:58:08,583 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
24 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462,
length=64341194
2011-06-30 22:58:08,583 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
2011-06-30 22:58:09,585 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
2011-06-30 22:58:11,171 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25978
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
2011-06-30 22:58:11,171 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
25 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064,
length=64083581
2011-06-30 22:58:11,171 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
2011-06-30 22:58:12,173 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
2011-06-30 22:58:14,655 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26044
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
2011-06-30 22:58:14,656 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
26 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989,
length=64093785
2011-06-30 22:58:14,656 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
2011-06-30 22:58:15,658 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
2011-06-30 22:58:17,914 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25712
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
2011-06-30 22:58:17,915 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
27 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493,
length=64115777
2011-06-30 22:58:17,915 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
2011-06-30 22:58:18,918 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
2011-06-30 22:58:20,762 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
2011-06-30 22:58:20,762 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
28 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468,
length=64628500
2011-06-30 22:58:20,762 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
2011-06-30 22:58:21,766 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
2011-06-30 22:58:23,158 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26131
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
2011-06-30 22:58:23,159 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
29 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048,
length=64522869
2011-06-30 22:58:23,159 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
2011-06-30 22:58:24,162 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
2011-06-30 22:58:25,593 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26159
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
2011-06-30 22:58:25,593 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
30 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226,
length=64127204
2011-06-30 22:58:25,593 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
2011-06-30 22:58:26,594 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
2011-06-30 22:58:28,023 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
2011-06-30 22:58:28,023 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
31 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208,
length=96405599
2011-06-30 22:58:28,023 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
2011-06-30 22:58:29,026 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
2011-06-30 22:58:31,603 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38922
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
2011-06-30 22:58:31,603 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
32 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723,
length=64219650
2011-06-30 22:58:31,603 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
2011-06-30 22:58:32,607 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
2011-06-30 22:58:33,924 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
2011-06-30 22:58:33,924 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
33 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548,
length=64734596
2011-06-30 22:58:33,924 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
2011-06-30 22:58:34,927 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
2011-06-30 22:58:37,455 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26330
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
2011-06-30 22:58:37,456 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
34 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138,
length=95982267
2011-06-30 22:58:37,456 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
2011-06-30 22:58:38,459 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
2011-06-30 22:58:40,216 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38754
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
2011-06-30 22:58:40,216 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
35 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661,
length=0
2011-06-30 22:58:40,216 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
2011-06-30 22:58:41,219 INFO org.apache.hadoop.hbase.util.FSUtils:
Finished lease recover attempt for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
2011-06-30 22:58:41,219 WARN
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
might be still open, length is 0
2011-06-30 22:58:42,146 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=23462
entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
2011-06-30 22:58:42,179 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
2011-06-30 22:58:42,185 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
2011-06-30 22:58:42,191 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
2011-06-30 22:58:42,196 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
2011-06-30 22:58:42,203 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
2011-06-30 22:58:42,255 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
2011-06-30 22:58:42,270 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
2011-06-30 22:58:42,276 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
2011-06-30 22:58:42,292 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
2011-06-30 22:58:42,300 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
2011-06-30 22:58:42,306 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
2011-06-30 22:58:42,321 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
2011-06-30 22:58:42,329 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
2011-06-30 22:58:42,336 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
2011-06-30 22:58:42,342 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
2011-06-30 22:58:42,348 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
2011-06-30 22:58:42,354 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
2011-06-30 22:58:42,360 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
2011-06-30 22:58:42,366 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
2011-06-30 22:58:42,372 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
2011-06-30 22:58:42,378 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
2011-06-30 22:58:42,408 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
2011-06-30 22:58:42,414 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
2011-06-30 22:58:42,420 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
2011-06-30 22:58:42,426 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
2011-06-30 22:58:42,437 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
2011-06-30 22:58:42,444 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
2011-06-30 22:58:42,491 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
2011-06-30 22:58:42,504 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
2011-06-30 22:58:42,510 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
2011-06-30 22:58:42,516 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
2011-06-30 22:58:42,522 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
2011-06-30 22:58:42,528 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
2011-06-30 22:58:42,534 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
2011-06-30 22:58:42,540 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
2011-06-30 22:58:42,546 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for
split writer threads to finish
2011-06-30 22:58:49,942 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers
finished
2011-06-30 22:58:50,002 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
(wrote 62511 edits in 6859ms)
2011-06-30 22:58:50,014 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
(wrote 63273 edits in 9130ms)
2011-06-30 22:58:51,226 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
(wrote 63193 edits in 29376ms)
2011-06-30 22:58:51,232 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
(wrote 62727 edits in 16595ms)
2011-06-30 22:58:52,413 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
(wrote 62672 edits in 7177ms)
2011-06-30 22:58:52,461 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
(wrote 62636 edits in 52110ms)
2011-06-30 22:58:52,509 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
(wrote 62860 edits in 19402ms)
2011-06-30 22:58:52,515 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
(wrote 62718 edits in 12592ms)
2011-06-30 22:58:52,605 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
(wrote 62561 edits in 12576ms)
2011-06-30 22:58:52,670 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
(wrote 62860 edits in 9056ms)
2011-06-30 22:58:52,719 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
(wrote 63328 edits in 8381ms)
2011-06-30 22:58:52,725 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
(wrote 62040 edits in 7900ms)
2011-06-30 22:58:52,731 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
(wrote 62397 edits in 19988ms)
2011-06-30 22:58:52,779 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
(wrote 62952 edits in 21391ms)
2011-06-30 22:58:52,785 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
(wrote 62765 edits in 15781ms)
2011-06-30 22:58:52,785 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
splitting completed in 121163 ms for
hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
2011-06-30 22:58:52,893 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Reassigning 16 region(s) that
xmlqa-clv4.sh.intel.com,50820,1309421874765 was carrying (skipping 0
regions(s) that are already in transition)
2011-06-30 22:58:52,894 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 6a22ac864ca20a7f34b8822b6b092697 with OFFLINE state
2011-06-30 22:58:52,896 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.,
src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:52,896 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
to hadoop05.sh.intel.com,50820,1309421836726
2011-06-30 22:58:52,898 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 625da13f7c9f8d2a6db3e7bd892de061 with OFFLINE state
2011-06-30 22:58:52,900 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.,
src=, dest=harpertown083.sh.intel.com,50820,1309421853805; 10
(online=10, exclude=null) available servers
2011-06-30 22:58:52,900 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
to harpertown083.sh.intel.com,50820,1309421853805
2011-06-30 22:58:52,900 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop05.sh.intel.com,50820,1309421836726,
region=6a22ac864ca20a7f34b8822b6b092697
2011-06-30 22:58:52,902 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 32c8d35a95c6ac4f399c17dfa04066b1 with OFFLINE state
2011-06-30 22:58:52,904 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.,
src=, dest=xmlqa-clv6.sh.intel.com,50820,1309421880784; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:52,904 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
to xmlqa-clv6.sh.intel.com,50820,1309421880784
2011-06-30 22:58:52,904 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=harpertown083.sh.intel.com,50820,1309421853805,
region=625da13f7c9f8d2a6db3e7bd892de061
2011-06-30 22:58:52,906 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 0ea99d0d995b95e8ea07441d30aa5545 with OFFLINE state
2011-06-30 22:58:52,935 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.,
src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:52,935 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
to hadoop05.sh.intel.com,50820,1309421836726
2011-06-30 22:58:52,936 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=xmlqa-clv6.sh.intel.com,50820,1309421880784,
region=32c8d35a95c6ac4f399c17dfa04066b1
2011-06-30 22:58:52,937 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for d5a18e344d77bdccb568fd59623fb566 with OFFLINE state
2011-06-30 22:58:52,939 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.,
src=, dest=hbase01.sh.intel.com,50820,1309421885586; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:52,939 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
to hbase01.sh.intel.com,50820,1309421885586
2011-06-30 22:58:52,940 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop05.sh.intel.com,50820,1309421836726,
region=0ea99d0d995b95e8ea07441d30aa5545
2011-06-30 22:58:52,941 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for ed60ec735e30db1d99290995eb1cd2d7 with OFFLINE state
2011-06-30 22:58:52,943 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hbase01.sh.intel.com,50820,1309421885586,
region=d5a18e344d77bdccb568fd59623fb566
2011-06-30 22:58:52,943 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.,
src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:52,943 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
to hadoop01.sh.intel.com,50820,1309421825940
2011-06-30 22:58:52,945 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
and split region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
checking daughter presence
2011-06-30 22:58:52,946 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop01.sh.intel.com,50820,1309421825940,
region=ed60ec735e30db1d99290995eb1cd2d7
2011-06-30 22:58:53,005 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
present
2011-06-30 22:58:53,065 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
present
2011-06-30 22:58:53,065 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 3e0c81217d4b47920e11960f1816efae with OFFLINE state
2011-06-30 22:58:53,067 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.,
src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,067 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
to hadoop01.sh.intel.com,50820,1309421825940
2011-06-30 22:58:53,070 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for e4a2320bc8ff30fee1a9b20267b399cf with OFFLINE state
2011-06-30 22:58:53,072 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop01.sh.intel.com,50820,1309421825940,
region=3e0c81217d4b47920e11960f1816efae
2011-06-30 22:58:53,072 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.,
src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,072 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
to hadoop06.sh.intel.com,50820,1309421846110
2011-06-30 22:58:53,083 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 5a4a923e81c6127eb7f3ed61e5774063 with OFFLINE state
2011-06-30 22:58:53,085 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.,
src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,085 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
to ditest02.sh.intel.com,50820,1309421803646
2011-06-30 22:58:53,085 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop06.sh.intel.com,50820,1309421846110,
region=e4a2320bc8ff30fee1a9b20267b399cf
2011-06-30 22:58:53,086 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for d673806c5348d8517dcdb3392e5f5b21 with OFFLINE state
2011-06-30 22:58:53,087 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.,
src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,087 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
to hadoop06.sh.intel.com,50820,1309421846110
2011-06-30 22:58:53,088 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=ditest02.sh.intel.com,50820,1309421803646,
region=5a4a923e81c6127eb7f3ed61e5774063
2011-06-30 22:58:53,090 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 7cfa5c93f2d46d01c8f60ff934053ad1 with OFFLINE state
2011-06-30 22:58:53,117 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.,
src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,117 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
to hadoop05.sh.intel.com,50820,1309421836726
2011-06-30 22:58:53,117 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop06.sh.intel.com,50820,1309421846110,
region=d673806c5348d8517dcdb3392e5f5b21
2011-06-30 22:58:53,120 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 84c6f6ce13ffbeb3eb83c765fef754f3 with OFFLINE state
2011-06-30 22:58:53,126 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.,
src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,126 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
to ditest02.sh.intel.com,50820,1309421803646
2011-06-30 22:58:53,127 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop05.sh.intel.com,50820,1309421836726,
region=7cfa5c93f2d46d01c8f60ff934053ad1
2011-06-30 22:58:53,129 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for c94dc395183fd5a2286ec823b194b517 with OFFLINE state
2011-06-30 22:58:53,143 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.,
src=, dest=ditest01.sh.intel.com,50820,1309421799506; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,143 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
to ditest01.sh.intel.com,50820,1309421799506
2011-06-30 22:58:53,143 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=ditest02.sh.intel.com,50820,1309421803646,
region=84c6f6ce13ffbeb3eb83c765fef754f3
2011-06-30 22:58:53,149 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=ditest01.sh.intel.com,50820,1309421799506,
region=c94dc395183fd5a2286ec823b194b517
2011-06-30 22:58:53,151 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for 0dd9c11896eca61505861a9e49f2fef3 with OFFLINE state
2011-06-30 22:58:53,153 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.,
src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
exclude=null) available servers
2011-06-30 22:58:53,153 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
to ditest02.sh.intel.com,50820,1309421803646
2011-06-30 22:58:53,156 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
for d9adac25d8b83a3d4c252acaf5ba17af with OFFLINE state
2011-06-30 22:58:53,157 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
so generated a random one;
hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.,
src=, dest=harpertown0816.sh.intel.com,50820,1309421864349; 10
(online=10, exclude=null) available servers
2011-06-30 22:58:53,157 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
to harpertown0816.sh.intel.com,50820,1309421864349
2011-06-30 22:58:53,158 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=ditest02.sh.intel.com,50820,1309421803646,
region=0dd9c11896eca61505861a9e49f2fef3
2011-06-30 22:58:53,159 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
processing of shutdown of xmlqa-clv4.sh.intel.com,50820,1309421874765
Thanks
Weihua
2011/7/4 Ted Yu <yu...@gmail.com>:
> In the future, please direct questions on cdh releases to
> cdh-dev@cloudera.org
> You may cc dev@hbase.apache.org
>
> There is more than one minute difference between master and RS logs.
> Which one of the daughter regions didn't come online ?
>
> Cheers
>
> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com> wrote:
>
>> The HBase version we are using is CDH3U0.
>>
>> Thanks
>> Weihua
>>
>> 2011/7/4 Weihua JIANG <we...@gmail.com>:
>> > Hi all,
>> >
>> > We encountered a problem about region not onlining. A region is
>> > splitted by a closing RS and then this RS down. It seems master has
>> > known this split but it doesn't tried to make it online. Log from
>> > master
>> > 2011-06-30 22:58:52,945 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>> > and split region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>> > checking daughter presence
>> > 2011-06-30 22:58:52,946 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> > 2011-06-30 22:58:53,005 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > present
>> > 2011-06-30 22:58:53,065 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > present
>> >
>> > Log from RS is:
>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820 caught:
>> > java.nio.channels.ClosedChannelException
>> > at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>> >
>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820: exiting
>> > 2011-06-30 22:57:05,767 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Closed zookeeper sessionid=0x130ba69074900b4
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x130ba69074900b4 closed
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>> > EventThread shut down
>> > 2011-06-30 22:57:05,857 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > 2011-06-30 22:57:05,863 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > 2011-06-30 22:57:05,911 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>> > in META
>> > 2011-06-30 22:57:05,942 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,943 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>> > updated, and report to master.
>> >
>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>> > new regions:
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>> > Split took 1mins, 12sec
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Split Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Large Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Small Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>> > exiting
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > starting; hbase.shutdown.hook=true;
>> > fsShutdownHook=Thread[Thread-15,5,main]
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> > hook
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> > shutdown hook thread.
>> > 2011-06-30 22:57:06,196 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > finished.
>> >
>> >
>> > Thanks
>> > Weihua
>> >
>>
>
Re: Region not online after split by a closing RS
Posted by Weihua JIANG <we...@gmail.com>.
Both daughter regions are not online.
Thanks
Weihua
2011/7/4 Ted Yu <yu...@gmail.com>:
> In the future, please direct questions on cdh releases to
> cdh-dev@cloudera.org
> You may cc dev@hbase.apache.org
>
> There is more than one minute difference between master and RS logs.
> Which one of the daughter regions didn't come online ?
>
> Cheers
>
> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com> wrote:
>
>> The HBase version we are using is CDH3U0.
>>
>> Thanks
>> Weihua
>>
>> 2011/7/4 Weihua JIANG <we...@gmail.com>:
>> > Hi all,
>> >
>> > We encountered a problem about region not onlining. A region is
>> > splitted by a closing RS and then this RS down. It seems master has
>> > known this split but it doesn't tried to make it online. Log from
>> > master
>> > 2011-06-30 22:58:52,945 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>> > and split region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>> > checking daughter presence
>> > 2011-06-30 22:58:52,946 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> > 2011-06-30 22:58:53,005 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > present
>> > 2011-06-30 22:58:53,065 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > present
>> >
>> > Log from RS is:
>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820 caught:
>> > java.nio.channels.ClosedChannelException
>> > at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> > at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>> >
>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820: exiting
>> > 2011-06-30 22:57:05,767 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Closed zookeeper sessionid=0x130ba69074900b4
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x130ba69074900b4 closed
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>> > EventThread shut down
>> > 2011-06-30 22:57:05,857 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > 2011-06-30 22:57:05,863 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > 2011-06-30 22:57:05,911 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>> > in META
>> > 2011-06-30 22:57:05,942 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,943 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>> > updated, and report to master.
>> >
>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>> > new regions:
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>> > Split took 1mins, 12sec
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Split Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Large Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Small Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>> > exiting
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > starting; hbase.shutdown.hook=true;
>> > fsShutdownHook=Thread[Thread-15,5,main]
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> > hook
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> > shutdown hook thread.
>> > 2011-06-30 22:57:06,196 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > finished.
>> >
>> >
>> > Thanks
>> > Weihua
>> >
>>
>
Re: Region not online after split by a closing RS
Posted by Ted Yu <yu...@gmail.com>.
In the future, please direct questions on cdh releases to
cdh-dev@cloudera.org
You may cc dev@hbase.apache.org
There is more than one minute difference between master and RS logs.
Which one of the daughter regions didn't come online ?
Cheers
On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <we...@gmail.com> wrote:
> The HBase version we are using is CDH3U0.
>
> Thanks
> Weihua
>
> 2011/7/4 Weihua JIANG <we...@gmail.com>:
> > Hi all,
> >
> > We encountered a problem about region not onlining. A region is
> > splitted by a closing RS and then this RS down. It seems master has
> > known this split but it doesn't tried to make it online. Log from
> > master
> > 2011-06-30 22:58:52,945 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> > and split region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> > checking daughter presence
> > 2011-06-30 22:58:52,946 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop01.sh.intel.com,50820,1309421825940,
> > region=ed60ec735e30db1d99290995eb1cd2d7
> > 2011-06-30 22:58:53,005 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > present
> > 2011-06-30 22:58:53,065 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > present
> >
> > Log from RS is:
> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> > Server handler 73 on 50820 caught:
> > java.nio.channels.ClosedChannelException
> > at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
> >
> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server handler 73 on 50820: exiting
> > 2011-06-30 22:57:05,767 INFO
> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
> > leases
> > 2011-06-30 22:57:05,768 INFO
> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
> > leases
> > 2011-06-30 22:57:05,768 INFO
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Closed zookeeper sessionid=0x130ba69074900b4
> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x130ba69074900b4 closed
> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
> > EventThread shut down
> > 2011-06-30 22:57:05,857 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > 2011-06-30 22:57:05,863 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > 2011-06-30 22:57:05,911 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
> > in META
> > 2011-06-30 22:57:05,942 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > in region .META.,,1, serverInfo=null
> > 2011-06-30 22:57:05,943 INFO
> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> > daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > because stopping=false, stopped=true
> > 2011-06-30 22:57:05,950 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > in region .META.,,1, serverInfo=null
> > 2011-06-30 22:57:05,950 INFO
> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> > daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > because stopping=false, stopped=true
> > 2011-06-30 22:57:06,004 INFO
> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> > updated, and report to master.
> >
> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
> > new regions:
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
> > Split took 1mins, 12sec
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Split Thread to finish...
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Large Compaction Thread to finish...
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Small Compaction Thread to finish...
> > 2011-06-30 22:57:06,004 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
> > exiting
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > starting; hbase.shutdown.hook=true;
> > fsShutdownHook=Thread[Thread-15,5,main]
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> > hook
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> > shutdown hook thread.
> > 2011-06-30 22:57:06,196 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > finished.
> >
> >
> > Thanks
> > Weihua
> >
>
Re: Region not online after split by a closing RS
Posted by Weihua JIANG <we...@gmail.com>.
The HBase version we are using is CDH3U0.
Thanks
Weihua
2011/7/4 Weihua JIANG <we...@gmail.com>:
> Hi all,
>
> We encountered a problem about region not onlining. A region is
> splitted by a closing RS and then this RS down. It seems master has
> known this split but it doesn't tried to make it online. Log from
> master
> 2011-06-30 22:58:52,945 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> and split region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> checking daughter presence
> 2011-06-30 22:58:52,946 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:58:53,005 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> present
>
> Log from RS is:
> 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 73 on 50820 caught:
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 73 on 50820: exiting
> 2011-06-30 22:57:05,767 INFO
> org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
> leases
> 2011-06-30 22:57:05,768 INFO
> org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
> leases
> 2011-06-30 22:57:05,768 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Closed zookeeper sessionid=0x130ba69074900b4
> 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x130ba69074900b4 closed
> 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
> 2011-06-30 22:57:05,857 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> 2011-06-30 22:57:05,863 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> 2011-06-30 22:57:05,911 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
> in META
> 2011-06-30 22:57:05,942 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> in region .META.,,1, serverInfo=null
> 2011-06-30 22:57:05,943 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> because stopping=false, stopped=true
> 2011-06-30 22:57:05,950 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> in region .META.,,1, serverInfo=null
> 2011-06-30 22:57:05,950 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> because stopping=false, stopped=true
> 2011-06-30 22:57:06,004 INFO
> org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> updated, and report to master.
> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
> new regions: CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
> Split took 1mins, 12sec
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Split Thread to finish...
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Large Compaction Thread to finish...
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Small Compaction Thread to finish...
> 2011-06-30 22:57:06,004 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
> exiting
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> starting; hbase.shutdown.hook=true;
> fsShutdownHook=Thread[Thread-15,5,main]
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> hook
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> shutdown hook thread.
> 2011-06-30 22:57:06,196 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> finished.
>
>
> Thanks
> Weihua
>