You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/07/19 18:45:32 UTC
[jira] Commented: (HBASE-15) [hbase] Could not complete hdfs write
out to flush file forcing regionserver restart
[ https://issues.apache.org/jira/browse/HBASE-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615018#action_12615018 ]
Andrew Purtell commented on HBASE-15:
-------------------------------------
>From hbase-users@
From: "Renaud Delbru" <re...@deri.org>
To: hbase-user@hadoop.apache.org
Saturday, July 19, 2008 7:46 AM
The cluster is currently composed of 3 nodes. The cluster is also used
for executing hadoop jobs. One of the node acts as an hadoop master, an
HMaster and an HRegionServer.
The current settings are: Xmx=1024M for the Hadoop Master, Xmx=2048M for
the HMaster, Xmx=2048M for each HRegionServer.
The inserts are somehow randomised. We have 24 hadoop jobs uploading
various documents. The row key is the URL of the document, and each
hadoop task has to upload a range of URLs (for example,
http://my-url/AAA - http://my-url/BBB).
Thus, I assume that each hadoop tasks will hit a different region and
that the insert load is distributed among the nodes.
Thanks.
--
Renaud Delbru
Jean-Daniel Cryans wrote:
> Ok then, at least having the nodes synchronized keeps you safe from other
> problems. By the way, how many nodes do you have?
>
> One other thing to check is your writing pattern. If you insert data with
> sequential row keys, an example would be designing your row keys as an
> incremented primary key equivalent but left padded with zeros, these inserts
> will always hit on a single region thus the load will not be distributed. If
> it's your case, try randomizing the inserts.
>
> I will also look at your logs monday morning.
>
> Thx,
>
> J-D
>
> On Sat, Jul 19, 2008 at 5:47 AM, Renaud Delbru <re...@deri.org>
> wrote:
>
>
>> Hi Jean-Daniel,
>>
>> I have synchronised all the nodes to the same ntp server yesterday.
>> This morning, another regionserver went down for the same reason (log
>> attached).
>>
>> Thanks.
>> --
>> Renaud Delbru
>>
>>
>>
>> Jean-Daniel Cryans wrote:
>>
>>
>>> Renaud,
>>>
>>> Yes, the impact is important. Try fixing it first because typically heavy
>>> write load problems only occur when there is a very few number of nodes
>>> and
>>> regions. We have seen this many times.
>>>
>>> Happy fixing!
>>>
>>> J-D
>>>
>>> On Fri, Jul 18, 2008 at 5:20 PM, Renaud Delbru <re...@deri.org>
>>> wrote:
>>>
>>>
>>>
>>>
>>>> Hi Jean-Daniel,
>>>>
>>>> my nodes clocks are desynchronised (each node has a different time).
>>>> Could
>>>> it have an impact on HBase ?
>>>>
>>>> At the moment of the failure, the node had about 55 regions (with
>>>> hbase.hregion.max.filesize = 128M).
>>>>
>>>> Thanks.
>>>> --
>>>> Renaud Delbru
>>>>
>>>> As One node is at
>>>>
>>>> Jean-Daniel Cryans wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> Renaud,
>>>>>
>>>>> Just to make sure, are all your nodes clocks synchronized? Also, at the
>>>>> moment of that failure, how many regions did the table have?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> J-D
>>>>>
>>>>> On Fri, Jul 18, 2008 at 5:11 AM, Renaud Delbru <re...@deri.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using HBase 0.2.0-dev, Hudson Build #208.
>>>>>> I am experiencing a problem with HBase that looks like the issue
>>>>>> https://issues.apache.org/jira/browse/HBASE-15.
>>>>>> During an upload intensive task, one of the regionserver went down to
>>>>>> force rerun of hlog.
>>>>>>
>>>>>> This is not the first time we have experienced this problem, it
>>>>>> happened
>>>>>> already on another cluster node a few days ago, in the same settings.
>>>>>>
>>>>>> I have attached the regionserver log. I don't notice something
>>>>>> particular in the hadoop datanode log.
>>>>>>
>>>>>> Thanks in advance for any advices.
>>>>>> --
>>>>>> Renaud Delbru
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2008-07-18 03:40:51,861 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
>>>>>> entries: 30008
>>>>>> 2008-07-18 03:40:52,073 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>>> New
>>>>>> log writer created at
>>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216348851959
>>>>>> 2008-07-18 03:43:36,257 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
>>>>>> entries: 30008
>>>>>> 2008-07-18 03:43:37,020 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>>> New
>>>>>> log writer created at
>>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349016755
>>>>>> 2008-07-18 03:45:12,963 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18
>>>>>>
>>>>>>> 03:45:58,868
>>>>>>>
>>>>>> WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30837ms,
>>>>>>
>>>>>> ten times longer than scheduled: 3000
>>>>>> 2008-07-18 03:55:57,783 WARN org.apache.hadoop.hbase.util.Sleeper: We
>>>>>> slept
>>>>>> 583701ms, ten times longer than scheduled: 3000
>>>>>> 2008-07-18 03:55:57,787 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
>>>>>> entries: 21011
>>>>>> 2008-07-18 03:55:58,410 WARN org.apache.hadoop.dfs.DFSClient: Exception
>>>>>> while reading from blk_-6424749704096696952 of
>>>>>>
>>>>>> /hbase/page-repository/942413588/field/mapfiles/7570604533190594376/data
>>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from
>>>>>> inputStream
>>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
>>>>>> at
>>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388)
>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
>>>>>> at
>>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:1682)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1799)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828)
>>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84)
>>>>>>
>>>>>> 2008-07-18 03:55:58,966 WARN org.apache.hadoop.dfs.DFSClient: Exception
>>>>>> while reading from blk_7305122468419183134 of
>>>>>>
>>>>>> /hbase/page-repository/942413588/field/mapfiles/4454700123966493614/data
>>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from
>>>>>> inputStream
>>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
>>>>>> at
>>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388)
>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
>>>>>> at
>>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1709)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1745)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1831)
>>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84)
>>>>>>
>>>>>> 2008-07-18 03:56:05,305 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>>> New
>>>>>> log writer created at
>>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349764848
>>>>>> 2008-07-18 03:57:30,244 ERROR
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening
>>>>>> scanner
>>>>>> (fsOk: true)
>>>>>> java.io.IOException: HStoreScanner failed construction
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186)
>>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>>>>> response
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
>>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
>>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365)
>>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677)
>>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66)
>>>>>> ... 10 more
>>>>>> 2008-07-18 03:58:01,895 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 7 on 60020, call openScanner([B@9dd0d92, [[B@fa5f14d,
>>>>>> [B@5784fe41,
>>>>>> 9223372036854775807, null) from 140.203.154.174:48958: error:
>>>>>> java.io.IOException: HStoreScanner failed construction
>>>>>> java.io.IOException: HStoreScanner failed construction
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186)
>>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>>>>> response
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
>>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
>>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365)
>>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677)
>>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66)
>>>>>> ... 10 more
>>>>>> 2008-07-18 03:58:06,728 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> compaction completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401 in 12mins,
>>>>>> 53sec
>>>>>> 2008-07-18 03:58:06,728 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18
>>>>>>
>>>>>>> 03:58:06,729
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482 in 0sec
>>>>>> 2008-07-18 03:58:06,729 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18
>>>>>>
>>>>>>> 03:58:06,730
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425 in 0sec
>>>>>> 2008-07-18 03:58:06,730 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18
>>>>>>
>>>>>>> 03:58:06,760
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539in0sec
>>>>>> 2008-07-18 03:58:06,760 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18
>>>>>>
>>>>>>> 03:58:06,760
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811in0sec
>>>>>> 2008-07-18 03:58:06,761 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18
>>>>>>
>>>>>>> 03:58:06,761
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911 in 0sec
>>>>>> 2008-07-18 03:58:06,761 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18
>>>>>>
>>>>>>> 03:58:06,777
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411in0sec
>>>>>> 2008-07-18 03:58:06,777 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18
>>>>>>
>>>>>>> 03:58:06,800
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137 in 0sec
>>>>>> 2008-07-18 03:58:06,800 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18
>>>>>>
>>>>>>> 03:58:06,826
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663in
>>>>>> 0sec
>>>>>> 2008-07-18 03:58:06,826 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18
>>>>>>
>>>>>>> 03:58:06,916
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814in0sec
>>>>>> 2008-07-18 03:58:06,916 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://sws.geonames.org/3025251/,1215824259058
>>>>>> 2008-07-18 <http://sws.geonames.org/3025251/,12158242590582008-07-18
>>>>>>
>>>>>>> 03:58:06,972
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://sws.geonames.org/3025251/,1215824259058 in 0sec
>>>>>> 2008-07-18 03:58:06,973 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18
>>>>>>
>>>>>>> 03:58:06,973
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046in
>>>>>> 0sec
>>>>>> 2008-07-18 03:58:06,973 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18
>>>>>>
>>>>>>> 03:58:15,979
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
>>>>>> hlog. Number of entries: 30009
>>>>>> 2008-07-18 03:59:32,093 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> compaction completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565 in 1mins,
>>>>>> 25sec
>>>>>> 2008-07-18 03:59:32,093 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330
>>>>>> 2008-07-18<
>>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18
>>>>>>
>>>>>>> 03:59:32,134
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330 in 0sec
>>>>>> 2008-07-18 03:59:32,134 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18
>>>>>>
>>>>>>> 04:00:45,884
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009 in
>>>>>> 1mins,
>>>>>> 13sec
>>>>>> 2008-07-18 04:00:45,885 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18
>>>>>>
>>>>>>> 04:00:45,887
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943 in 0sec
>>>>>> 2008-07-18 04:00:45,887 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18
>>>>>>
>>>>>>> 04:00:45,892
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842in0sec
>>>>>> 2008-07-18 04:00:45,892 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18
>>>>>>
>>>>>>> 04:00:45,893
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438 in 0sec
>>>>>> 2008-07-18 04:00:45,893 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791
>>>>>> 2008-07-18<
>>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18
>>>>>>
>>>>>>> 04:00:45,895
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791 in 0sec
>>>>>> 2008-07-18 04:00:45,895 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://sws.geonames.org/1081116/,1215823957458
>>>>>> 2008-07-18 <http://sws.geonames.org/1081116/,12158239574582008-07-18
>>>>>>
>>>>>>> 04:00:45,895
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://sws.geonames.org/1081116/,1215823957458 in 0sec
>>>>>> 2008-07-18 04:00:45,896 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/Allomeric,1216039960663
>>>>>> 2008-07-18 <
>>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18
>>>>>>
>>>>>>> 04:00:45,935
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/Allomeric,1216039960663 in 0sec
>>>>>> 2008-07-18 04:00:45,935 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18
>>>>>>
>>>>>>> 04:00:45,993
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259in0sec
>>>>>> 2008-07-18 04:00:45,994 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18
>>>>>>
>>>>>>> 04:00:46,004
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332 in
>>>>>> 0sec
>>>>>> 2008-07-18 04:00:46,005 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18
>>>>>>
>>>>>>> 04:00:46,005
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>> completed on region page-repository,
>>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135 in
>>>>>> 0sec
>>>>>> 2008-07-18 04:00:46,005 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> starting compaction on region page-repository,
>>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856
>>>>>> 2008-07-18<
>>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18
>>>>>>
>>>>>>> 04:00:46,007
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
>>>>>>
>>>>>> completed on region page-repository,
>>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856 in 0sec
>>>>>> 2008-07-18 04:02:16,798 FATAL
>>>>>> org.apache.hadoop.hbase.regionserver.Flusher:
>>>>>> Replay of hlog required. Forcing server restart
>>>>>> org.apache.hadoop.hbase.DroppedSnapshotException
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040)
>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93)
>>>>>> Caused by: org.apache.hadoop.ipc.RemoteException:
>>>>>> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create
>>>>>> file
>>>>>>
>>>>>>
>>>>>> /hbase/page-repository/1103034897/field/mapfiles/2990615865592052391/index
>>>>>> for DFSClient_716087201 on client 140.203.154.176 because current
>>>>>> leaseholder is trying to recreate file.
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:1010)
>>>>>> at
>>>>>> org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:967)
>>>>>> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:269)
>>>>>> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>>
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
>>>>>> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2189)
>>>>>> at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:479)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:138)
>>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:508)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1208)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:402)
>>>>>> at
>>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:307)
>>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:159)
>>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:133)
>>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:125)
>>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:110)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseWriter.<init>(HStoreFile.java:706)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Writer.<init>(HStoreFile.java:810)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getWriter(HStoreFile.java:453)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:706)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030)
>>>>>> ... 3 more
>>>>>> 2008-07-18 04:02:16,798 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.Flusher:
>>>>>> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
>>>>>> 2008-07-18 04:02:16,807 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 1 on 60020, call batchUpdate([B@167e3c87, row =>
>>>>>> http://dbpedia.org/resource/Jurrette_Sindelar, {column => field:class,
>>>>>> value => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.174:50719: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,847 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 9 on 60020, call batchUpdate([B@525cfe10, row =>
>>>>>> http://dbpedia.org/resource/Josef_Cyril_Sychra, {column =>
>>>>>> field:class,
>>>>>> value => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.175:48121: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,848 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 0 on 60020, call batchUpdate([B@572bf6ae, row =>
>>>>>> http://dbpedia.org/resource/John_Johnston_%28Royal_Household%29,
>>>>>> {column
>>>>>> => field:class, value => '...', column => field:data-source, value =>
>>>>>> '...',
>>>>>> column => field:domain, value => '...', column =>
>>>>>> field:explicit-content,
>>>>>> value => '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.175:48348: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,905 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 4 on 60020, call batchUpdate([B@27320a48, row =>
>>>>>> http://dbpedia.org/resource/Karim_Benkhirane, {column => field:class,
>>>>>> value => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.175:48330: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,926 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 4 on 60020, call batchUpdate([B@59c7d8c6, row =>
>>>>>> http://dbpedia.org/resource/Fort_Pierce_%28FL%29, {column =>
>>>>>> field:class,
>>>>>> value => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.176:58334: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,959 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 8 on 60020, call batchUpdate([B@4f68dd36, row =>
>>>>>> http://dbpedia.org/resource/Hendrik_Speuy, {column => field:class,
>>>>>> value
>>>>>> => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.175:48357: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:16,979 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 3 on 60020, call batchUpdate([B@596c7f5f, row =>
>>>>>> http://dbpedia.org/resource/Harry_Stuck_Sullivan, {column =>
>>>>>> field:class,
>>>>>> value => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.176:58163: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:17,147 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 2 on 60020, call batchUpdate([B@23067cee, row =>
>>>>>> http://dbpedia.org/resource/High_Littleton, {column => field:class,
>>>>>> value
>>>>>> => '...', column => field:data-source, value => '...', column =>
>>>>>> field:domain, value => '...', column => field:explicit-content, value
>>>>>> =>
>>>>>> '...', column => field:format, value => '...', column =>
>>>>>> field:implicit-content, value => '...', column => field:label, value =>
>>>>>> '...', column => field:length, value => '...', column =>
>>>>>> field:ontology,
>>>>>> value => '...', column => field:size, value => '...', column =>
>>>>>> field:timestamp, value => '...'}) from 140.203.154.174:50952: error:
>>>>>> java.io.IOException: Server not running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
>>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:17,180 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>>> New
>>>>>> log writer created at
>>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216350137145
>>>>>> 2008-07-18 04:02:17,181 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
>>>>>> 2008-07-18 04:02:17,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 7 on 60020, call getClosestRowBefore([B@20d09fed, [B@60b3916f)
>>>>>> from 140.203.154.175:48464: error: java.io.IOException: Server not
>>>>>> running
>>>>>> java.io.IOException: Server not running
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1092)
>>>>>> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>>>> 2008-07-18 04:02:17,507 INFO org.apache.hadoop.ipc.Server: Stopping
>>>>>> server
>>>>>> on 60020
>>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 1 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 5 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,587 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 0 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 2 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 9 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 8 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 7 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 6 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 4 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>>> handler 3 on 60020: exiting
>>>>>> 2008-07-18 04:02:17,607 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>>>>>> Server listener on 60020
>>>>>> 2008-07-18 04:02:17,691 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>>> 2008-07-18 04:02:17,784 INFO org.apache.hadoop.ipc.Server: Stopping IPC
>>>>>> Server Responder
>>>>>> 2008-07-18 04:02:17,902 INFO org.mortbay.util.ThreadedServer: Stopping
>>>>>> Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
>>>>>> 2008-07-18 04:02:18,179 INFO org.mortbay.http.SocketListener: Stopped
>>>>>> SocketListener on 0.0.0.0:60030
>>>>>> 2008-07-18 04:02:28,883 INFO org.mortbay.util.Container: Stopped
>>>>>> HttpContext[/static,/static]
>>>>>> 2008-07-18 04:02:31,116 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread
>>>>>> exiting
>>>>>> 2008-07-18 04:02:31,117 INFO org.mortbay.util.Container: Stopped
>>>>>> HttpContext[/logs,/logs]
>>>>>> 2008-07-18 04:02:31,117 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
>>>>>> regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
>>>>>> 2008-07-18 04:02:31,125 INFO org.mortbay.util.Container: Stopped
>>>>>> org.mortbay.jetty.servlet.WebApplicationHandler@53797795
>>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped
>>>>>> WebApplicationContext[/,/]
>>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped
>>>>>> org.mortbay.jetty.Server@b8c3679
>>>>>> 2008-07-18 04:02:34,309 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed
>>>>>> hlog
>>>>>> 2008-07-18 04:55:59,098 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -3929282808018198437 lease expired
>>>>>> 2008-07-18 04:56:00,258 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> 8694039619106515508 lease expired
>>>>>> 2008-07-18 04:56:02,068 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -6642383936144787497 lease expired
>>>>>> 2008-07-18 04:56:02,178 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -8674274402855984090 lease expired
>>>>>> 2008-07-18 04:56:02,858 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -8684213796880407868 lease expired
>>>>>> 2008-07-18 04:56:03,478 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -8003294377021166340 lease expired
>>>>>> 2008-07-18 04:56:05,298 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> 4741746108028229290 lease expired
>>>>>> 2008-07-18 04:56:05,418 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -3662383214249882020 lease expired
>>>>>> 2008-07-18 04:56:05,498 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>>>>> -5749397523507975233 lease expired
>>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases:
>>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases
>>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases:
>>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases
>>>>>> 2008-07-18 04:56:05,499 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> closed .META.,,1
>>>>>> 2008-07-18 04:56:05,499 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>>> closed page-repository,
>>>>>> http://dbpedia.org/resource/CAMBIA,1215970756532
>>>>>> 2008-07-18 <http://dbpedia.org/resource/CAMBIA,12159707565322008-07-18
>>>>>>
>>>>>>> 04:56:05,499
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18
>>>>>>
>>>>>>> 04:56:05,499
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://users.livejournal.com/_alicesprings/data/foaf,1215824861383
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://users.livejournal.com/_alicesprings/data/foaf,12158248613832008-07-18
>>>>>>
>>>>>>> 04:56:05,499
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,1216118586621
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,12161185866212008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://sws.geonames.org/1081116/,1215823957458
>>>>>> 2008-07-18 <http://sws.geonames.org/1081116/,12158239574582008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330
>>>>>> 2008-07-18<
>>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,1216118586621
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,12161185866212008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,1216109320961
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,12161093209612008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://purl.uniprot.org/go/0009140,1214582602882
>>>>>> 2008-07-18 <http://purl.uniprot.org/go/0009140,12145826028822008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18
>>>>>>
>>>>>>> 04:56:05,500
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://bio2rdf.org/omim:300357,1215825157318
>>>>>> 2008-07-18 <http://bio2rdf.org/omim:300357,12158251573182008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://alcidesfonseca.com/blog/35,1215825925595
>>>>>> 2008-07-18 <http://alcidesfonseca.com/blog/35,12158259255952008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/Allomeric,1216039960663
>>>>>> 2008-07-18 <
>>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791
>>>>>> 2008-07-18<
>>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Portland__Garden_Club,1216125563046
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Portland__Garden_Club,12161255630462008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/John_J._Bernet,1216109320961
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/John_J._Bernet,12161093209612008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18
>>>>>>
>>>>>>> 04:56:05,501
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://sws.geonames.org/3025251/,1215824259058
>>>>>> 2008-07-18 <http://sws.geonames.org/3025251/,12158242590582008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Billy_Forester,1216165245424
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Billy_Forester,12161652454242008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/Snatcho,1216263402275
>>>>>> 2008-07-18 <
>>>>>> http://dbpedia.org/resource/Snatcho,12162634022752008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Template:superheroe,1216140016580
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Template:superheroe,12161400165802008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18
>>>>>>
>>>>>>> 04:56:05,502
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/Bob_Errey,1216165245424
>>>>>> 2008-07-18 <
>>>>>> http://dbpedia.org/resource/Bob_Errey,12161652454242008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/The_Way_To_Fight,1216138073477
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/The_Way_To_Fight,12161380734772008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,1215959690520
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,12159596905202008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/Syed_Koteb,1216140016580
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Syed_Koteb,12161400165802008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/1er_RCP,1215980565468
>>>>>> 2008-07-18 <
>>>>>> http://dbpedia.org/resource/1er_RCP,12159805654682008-07-18
>>>>>>
>>>>>>> 04:56:05,503
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://purl.uniprot.org/taxonomy/112557,1214582151856
>>>>>> 2008-07-18<
>>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Harry_Girvetz,1215910087698
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Harry_Girvetz,12159100876982008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,1215922127438
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,12159221274382008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,1215825925595
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,12158259255952008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Category:1948_in_law,1215896373168
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Category:1948_in_law,12158963731682008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://acm.rkbexplorer.com/id/person-94180,1215825860040
>>>>>> 2008-07-18<
>>>>>> http://acm.rkbexplorer.com/id/person-94180,12158258600402008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,1215823941299
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,12158239412992008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Trevor_Ashmore,1216138073477
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Trevor_Ashmore,12161380734772008-07-18
>>>>>>
>>>>>>> 04:56:05,504
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,1215929139300
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,12159291393002008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/1960-09-03,1215980565468
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/1960-09-03,12159805654682008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>>
>>>>>>
>>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842
>>>>>> 2008-07-18<
>>>>>>
>>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18
>>>>>>
>>>>>>> 04:56:05,505
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18
>>>>>>
>>>>>>> 04:56:05,506
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://sws.geonames.org/2701307/,1215823954578
>>>>>> 2008-07-18 <http://sws.geonames.org/2701307/,12158239545782008-07-18
>>>>>>
>>>>>>> 04:56:05,506
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,
>>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18
>>>>>>
>>>>>>> 04:56:05,506
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>> page-repository,http://dbpedia.org/resource/Angus_McKay,1215890852482
>>>>>> 2008-07-18<
>>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18
>>>>>>
>>>>>>> 04:56:05,506
>>>>>>>
>>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>> aborting server at: 140.203.154.176:60020
>>>>>> 2008-07-18 04:56:05,506 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>> regionserver/0:0:0:0:0:0:0:0:60020 exiting
>>>>>> 2008-07-18 04:56:05,561 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
>>>>>> thread.
>>>>>> 2008-07-18 04:56:05,562 INFO
>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
>>>>>> complete
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>> 2008-07-19 07:31:45,505 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
>> Exception: java.net.SocketTimeoutException: timed out waiting for rpc
>> response
>> 2008-07-19 07:31:45,506 WARN org.apache.hadoop.dfs.DFSClient: Error
>> Recovery for block null bad datanode[0]
>> 2008-07-19 07:31:54,868 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
>> Replay of hlog required. Forcing server restart
>> org.apache.hadoop.hbase.DroppedSnapshotException
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942)
>> at
>> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
>> at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93)
>> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>> Could not complete write to file
>> /hbase/page-repository/963177038/field/info/4591605661998875281 by
>> DFSClient_1009756630
>> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332)
>> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
>> at
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655)
>> at
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576)
>> at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
>> at
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
>> at
>> org.apache.hadoop.hbase.regionserver.HStoreFile.writeInfo(HStoreFile.java:350)
>> at
>> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:747)
>> at
>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030)
>> ... 3 more
>> 2008-07-19 07:31:54,868 INFO org.apache.hadoop.hbase.regionserver.Flusher:
>> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
>>
>>
>>
>
>
> [hbase] Could not complete hdfs write out to flush file forcing regionserver restart
> ------------------------------------------------------------------------------------
>
> Key: HBASE-15
> URL: https://issues.apache.org/jira/browse/HBASE-15
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: stack
> Priority: Trivial
> Attachments: delete-logging.patch
>
>
> I've spent some time looking into this issue but there are not enough clues in the logs to tell where the problem is. Here's what I know.
> Two region servers went down last night, a minute apart, during Paul Saab's 6hr run inserting 300million rows into hbase. The regionservers went down to force rerun of hlog and avoid possible data loss after a failure writing memory flushes to hdfs.
> Here is the lead up to the failed flush:
> ...
> 2007-11-28 22:40:02,231 INFO hbase.HRegionServer - MSG_REGION_OPEN : regionname: postlog,img149/4699/133lm0.jpg,1196318393738, startKey: <img149/4699/133lm0.jpg>, tableDesc: {name: postlog, families: {cookie:={name: cookie, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}, ip:={name: ip, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}
> 2007-11-28 22:40:02,242 DEBUG hbase.HStore - starting 1703405830/cookie (no reconstruction log)
> 2007-11-28 22:40:02,741 DEBUG hbase.HStore - maximum sequence id for hstore 1703405830/cookie is 29077708
> 2007-11-28 22:40:03,094 DEBUG hbase.HStore - starting 1703405830/ip (no reconstruction log)
> 2007-11-28 22:40:03,852 DEBUG hbase.HStore - maximum sequence id for hstore 1703405830/ip is 29077708
> 2007-11-28 22:40:04,138 DEBUG hbase.HRegion - Next sequence id for region postlog,img149/4699/133lm0.jpg,1196318393738 is 29077709
> 2007-11-28 22:40:04,141 INFO hbase.HRegion - region postlog,img149/4699/133lm0.jpg,1196318393738 available
> 2007-11-28 22:40:04,141 DEBUG hbase.HLog - changing sequence number from 21357623 to 29077709
> 2007-11-28 22:40:04,141 INFO hbase.HRegionServer - MSG_REGION_OPEN : regionname: postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739, startKey: <img149/7512/dscn4444lightenedfi3.jpg>, tableDesc: {name: postlog, families: {cookie:={name: cookie, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}, ip:={name: ip, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}
> 2007-11-28 22:40:04,145 DEBUG hbase.HStore - starting 376748222/cookie (no reconstruction log)
> 2007-11-28 22:40:04,223 DEBUG hbase.HStore - maximum sequence id for hstore 376748222/cookie is 29077708
> 2007-11-28 22:40:04,277 DEBUG hbase.HStore - starting 376748222/ip (no reconstruction log)
> 2007-11-28 22:40:04,353 DEBUG hbase.HStore - maximum sequence id for hstore 376748222/ip is 29077708
> 2007-11-28 22:40:04,699 DEBUG hbase.HRegion - Next sequence id for region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739 is 29077709
> 2007-11-28 22:40:04,701 INFO hbase.HRegion - region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739 available
> 2007-11-28 22:40:34,427 DEBUG hbase.HRegionServer - flushing region postlog,img143/1310/yashrk3.jpg,1196317258704
> 2007-11-28 22:40:34,428 DEBUG hbase.HRegion - Not flushing cache for region postlog,img143/1310/yashrk3.jpg,1196317258704: snapshotMemcaches() determined that there was nothing to do
> 2007-11-28 22:40:55,745 DEBUG hbase.HRegionServer - flushing region postlog,img142/8773/1001417zc4.jpg,1196317258703
> 2007-11-28 22:40:55,745 DEBUG hbase.HRegion - Not flushing cache for region postlog,img142/8773/1001417zc4.jpg,1196317258703: snapshotMemcaches() determined that there was nothing to do
> 2007-11-28 22:41:04,144 DEBUG hbase.HRegionServer - flushing region postlog,img149/4699/133lm0.jpg,1196318393738
> 2007-11-28 22:41:04,144 DEBUG hbase.HRegion - Started memcache flush for region postlog,img149/4699/133lm0.jpg,1196318393738. Size 74.7k
> 2007-11-28 22:41:04,764 DEBUG hbase.HStore - Added 1703405830/ip/610047924323344967 with sequence id 29081563 and size 53.8k
> 2007-11-28 22:41:04,902 DEBUG hbase.HStore - Added 1703405830/cookie/3147798053949544972 with sequence id 29081563 and size 41.3k
> 2007-11-28 22:41:04,902 DEBUG hbase.HRegion - Finished memcache flush for region postlog,img149/4699/133lm0.jpg,1196318393738 in 758ms, sequenceid=29081563
> 2007-11-28 22:41:04,902 DEBUG hbase.HStore - compaction for HStore postlog,img149/4699/133lm0.jpg,1196318393738/ip needed.
> 2007-11-28 22:41:04,903 DEBUG hbase.HRegion - 1703405830/ip needs compaction
> 2007-11-28 22:41:04,903 INFO hbase.HRegion - starting compaction on region postlog,img149/4699/133lm0.jpg,1196318393738
> 2007-11-28 22:41:04,903 DEBUG hbase.HStore - started compaction of 4 files in /hbase/compaction.dir/hregion_1703405830/ip
> 2007-11-28 22:41:04,905 DEBUG hbase.HRegionServer - flushing region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739
> 2007-11-28 22:41:04,906 DEBUG hbase.HRegion - Started memcache flush for region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739. Size 133.1k
> 2007-11-28 22:41:05,087 DEBUG hbase.HStore - Added 376748222/ip/8686640221458382286 with sequence id 29081604 and size 95.8k
> 2007-11-28 22:41:05,419 FATAL hbase.HRegionServer - Replay of hlog required. Forcing server restart
> org.apache.hadoop.hbase.DroppedSnapshotException: java.io.IOException: Could not complete write to file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data by DFSClient_-1010714652
> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:323)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at org.apache.hadoop.hbase.HRegion.internalFlushcache(HRegion.java:920)
> at org.apache.hadoop.hbase.HRegion.flushcache(HRegion.java:807)
> at org.apache.hadoop.hbase.HRegionServer$Flusher.run(HRegionServer.java:451)
> ...
> Over in the namenode, I see the following related to the deleted file:
> 2007-11-28 22:40:48,546 DEBUG dfs.StateChange - *DIR* NameNode.mkdirs: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,546 DEBUG dfs.StateChange - DIR* NameSystem.mkdirs: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,546 DEBUG dfs.StateChange - DIR* FSDirectory.mkdirs: created directory /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,552 DEBUG dfs.StateChange - *DIR* NameNode.create: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652 at XX.XX.XX.29
> 2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* NameSystem.startFile: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652 at XX.XX.XX.29
> 2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data is added to the file system
> 2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* NameSystem.startFile: add /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data to namespace for DFSClient_-1010714652
> 2007-11-28 22:40:48,557 DEBUG dfs.StateChange - *DIR* NameNode.create: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index for DFSClient_-1010714652 at XX.XX.XX.29
> 2007-11-28 22:40:48,557 DEBUG dfs.StateChange - DIR* NameSystem.startFile: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index for DFSClient_-1010714652 XX.XX.XX.29
> 2007-11-28 22:40:48,558 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index is added to the file system
> 2007-11-28 22:40:48,558 DEBUG dfs.StateChange - DIR* NameSystem.startFile: add /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index to namespace for DFSClient_-1010714652
> 2007-11-28 22:40:48,573 DEBUG dfs.StateChange - *BLOCK* NameNode.addBlock: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
> 2007-11-28 22:40:48,573 DEBUG dfs.StateChange - BLOCK* NameSystem.getAdditionalBlock: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
> 2007-11-28 22:40:48,573 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data with blk_-6398846109350112398 block is added to the in-memory file system
> 2007-11-28 22:40:48,573 INFO dfs.StateChange - BLOCK* NameSystem.allocateBlock: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data. blk_-6398846109350112398
> .....
> 2007-11-28 22:40:48,717 DEBUG dfs.StateChange - *DIR* NameNode.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* NameSystem.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* FSDirectory.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
> 2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* FSDirectory.unprotectedDelete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411 is removed
> ....
> 2007-11-28 22:40:48,826 DEBUG dfs.StateChange - *DIR* NameNode.complete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
> 2007-11-28 22:40:48,826 DEBUG dfs.StateChange - DIR* NameSystem.completeFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
> 2007-11-28 22:40:48,827 WARN dfs.StateChange - DIR* NameSystem.completeFile: failed to complete /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data because dir.getFileBlocks() is null and pendingFile is null
> Only delete I see is on compaction completion but reading code doesn't look like its possible for deletes to be confused.
> Will add extra logging to see if I can figure how the delete is coming about.
> (We've fixed a similar problem before when compactions were done at region level)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.