You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/01/13 19:18:00 UTC
[jira] Created: (HBASE-1125) "IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close"
"IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close"
-------------------------------------------------------------------------------------------------------
Key: HBASE-1125
URL: https://issues.apache.org/jira/browse/HBASE-1125
Project: Hadoop HBase
Issue Type: Bug
Environment: Andrew Purtell's cluster (.31) running current TRUNK
Reporter: stack
Priority: Blocker
Fix For: 0.19.0
Its at startup. Here is the regionserver side of things:
{code}
...
2009-01-13 06:56:59,060 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
2009-01-13 06:56:59,060 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region content,0d952ef7d6d31c81552d84ec99c61138,1231777275065/2063013531
2009-01-13 06:56:59,106 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/1731769154213178962, isReference=false, sequence id=8231740, length=255693, majorCompaction=false
2009-01-13 06:56:59,129 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/2908303503677637558, isReference=false, sequence id=8485218, length=193497, majorCompaction=false
2009-01-13 06:56:59,138 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4011669738439968964, isReference=false, sequence id=2657244, length=628824, majorCompaction=false
2009-01-13 06:56:59,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4385158897188164442, isReference=false, sequence id=7927662, length=277950, majorCompaction=false
2009-01-13 06:56:59,168 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5107758633254723438, isReference=false, sequence id=7687719, length=195595, majorCompaction=false
2009-01-13 06:56:59,172 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5309895098547950637, isReference=false, sequence id=8544018, length=34716, majorCompaction=false
2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/7261360197361933359, isReference=false, sequence id=7468745, length=192133, majorCompaction=false
2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 7 file(s) in hstore 2063013531/url, max sequence id 8544018
2009-01-13 06:56:59,416 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
2009-01-13 06:57:02,452 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
2009-01-13 06:57:05,458 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
2009-01-13 06:57:08,464 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
at java.lang.Thread.run(Thread.java:619)
2009-01-13 06:57:08,466 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
at java.lang.Thread.run(Thread.java:619)
2009-01-13 06:57:08,737 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) java.io.IOException: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
...
{code}
Here is master side of things:
{code}
2009-01-13 06:57:08,450 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,810e91fecfad126862a26774bf34df75,1231783565254 from XX.XX.XX.50:60020
2009-01-13 06:57:08,734 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
at java.lang.Thread.run(Thread.java:619)
from 10.30.94.33:60020
2009-01-13 06:57:08,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call regionServerReport(address: XX.XX.XX.33:60020, startcode: 1231829713428, load: (requests=0, regions=21, usedHeap=158, maxHeap=1774), [Lorg.apache.hadoop.hbase.HMsg;@17496e57, [Lorg.apache.hadoop.hbase.HRegionInfo;@55eb1db2) from 10.30.94.33:37051: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:285)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:569)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
{code}
This puts region in funky state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1125) "IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663392#action_12663392 ]
stack commented on HBASE-1125:
------------------------------
So, looks like as far as master was concerned, the region wasn't open yet. Here is the last thing we got about the problematic region:
{code}
2009-01-13 06:56:59,721 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065 from 10.30.94.33:60020
{code}
It was 'opening' but the open fails because of above.
Looks like state transitions need to allow above possibility.
> "IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close"
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1125
> URL: https://issues.apache.org/jira/browse/HBASE-1125
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: Andrew Purtell's cluster (.31) running current TRUNK
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Its at startup. Here is the regionserver side of things:
> {code}
> ...
> 2009-01-13 06:56:59,060 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> 2009-01-13 06:56:59,060 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region content,0d952ef7d6d31c81552d84ec99c61138,1231777275065/2063013531
> 2009-01-13 06:56:59,106 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/1731769154213178962, isReference=false, sequence id=8231740, length=255693, majorCompaction=false
> 2009-01-13 06:56:59,129 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/2908303503677637558, isReference=false, sequence id=8485218, length=193497, majorCompaction=false
> 2009-01-13 06:56:59,138 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4011669738439968964, isReference=false, sequence id=2657244, length=628824, majorCompaction=false
> 2009-01-13 06:56:59,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4385158897188164442, isReference=false, sequence id=7927662, length=277950, majorCompaction=false
> 2009-01-13 06:56:59,168 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5107758633254723438, isReference=false, sequence id=7687719, length=195595, majorCompaction=false
> 2009-01-13 06:56:59,172 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5309895098547950637, isReference=false, sequence id=8544018, length=34716, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/7261360197361933359, isReference=false, sequence id=7468745, length=192133, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 7 file(s) in hstore 2063013531/url, max sequence id 8544018
> 2009-01-13 06:56:59,416 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:02,452 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:05,458 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:08,464 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,466 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,737 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) java.io.IOException: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> ...
> {code}
> Here is master side of things:
> {code}
> 2009-01-13 06:57:08,450 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,810e91fecfad126862a26774bf34df75,1231783565254 from XX.XX.XX.50:60020
> 2009-01-13 06:57:08,734 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> from 10.30.94.33:60020
> 2009-01-13 06:57:08,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call regionServerReport(address: XX.XX.XX.33:60020, startcode: 1231829713428, load: (requests=0, regions=21, usedHeap=158, maxHeap=1774), [Lorg.apache.hadoop.hbase.HMsg;@17496e57, [Lorg.apache.hadoop.hbase.HRegionInfo;@55eb1db2) from 10.30.94.33:37051: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:285)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:569)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> This puts region in funky state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1125) "IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close"
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman resolved HBASE-1125.
----------------------------------
Resolution: Fixed
Patch committed.
> "IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close"
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1125
> URL: https://issues.apache.org/jira/browse/HBASE-1125
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: Andrew Purtell's cluster (.31) running current TRUNK
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Its at startup. Here is the regionserver side of things:
> {code}
> ...
> 2009-01-13 06:56:59,060 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> 2009-01-13 06:56:59,060 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region content,0d952ef7d6d31c81552d84ec99c61138,1231777275065/2063013531
> 2009-01-13 06:56:59,106 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/1731769154213178962, isReference=false, sequence id=8231740, length=255693, majorCompaction=false
> 2009-01-13 06:56:59,129 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/2908303503677637558, isReference=false, sequence id=8485218, length=193497, majorCompaction=false
> 2009-01-13 06:56:59,138 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4011669738439968964, isReference=false, sequence id=2657244, length=628824, majorCompaction=false
> 2009-01-13 06:56:59,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4385158897188164442, isReference=false, sequence id=7927662, length=277950, majorCompaction=false
> 2009-01-13 06:56:59,168 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5107758633254723438, isReference=false, sequence id=7687719, length=195595, majorCompaction=false
> 2009-01-13 06:56:59,172 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5309895098547950637, isReference=false, sequence id=8544018, length=34716, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/7261360197361933359, isReference=false, sequence id=7468745, length=192133, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 7 file(s) in hstore 2063013531/url, max sequence id 8544018
> 2009-01-13 06:56:59,416 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:02,452 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:05,458 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:08,464 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,466 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,737 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) java.io.IOException: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> ...
> {code}
> Here is master side of things:
> {code}
> 2009-01-13 06:57:08,450 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,810e91fecfad126862a26774bf34df75,1231783565254 from XX.XX.XX.50:60020
> 2009-01-13 06:57:08,734 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> from 10.30.94.33:60020
> 2009-01-13 06:57:08,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call regionServerReport(address: XX.XX.XX.33:60020, startcode: 1231829713428, load: (requests=0, regions=21, usedHeap=158, maxHeap=1774), [Lorg.apache.hadoop.hbase.HMsg;@17496e57, [Lorg.apache.hadoop.hbase.HRegionInfo;@55eb1db2) from 10.30.94.33:37051: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:285)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:569)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> This puts region in funky state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1125) "IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663408#action_12663408 ]
stack commented on HBASE-1125:
------------------------------
Here is a better picture of whats going on with the problematic region:
{code}
2009-01-13 06:56:53,697 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065 from XX.XX.XX.33:60020
2009-01-13 06:56:56,701 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065 from XX.XX.XX.33:60020
2009-01-13 06:56:59,721 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065 from XX.XX.XX.33:60020
2009-01-13 06:57:08,734 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
2009-01-13 06:57:08,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call regionServerReport(address: 10.30.94.33:60020, startcode: 1231829713428, load: (requests=0, regions=21, usedHeap=158, maxHeap=1774), [Lorg.apache.hadoop.hbase.HMsg;@17496e57, [Lorg.apache.hadoop.hbase.HRegionInfo;@55eb1db2) from XX.XX.XX.33:37051: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
...
{code}
Hereafter its hosed until A steps in and does manual fixup (I believe)
> "IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close"
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1125
> URL: https://issues.apache.org/jira/browse/HBASE-1125
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: Andrew Purtell's cluster (.31) running current TRUNK
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Its at startup. Here is the regionserver side of things:
> {code}
> ...
> 2009-01-13 06:56:59,060 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> 2009-01-13 06:56:59,060 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region content,0d952ef7d6d31c81552d84ec99c61138,1231777275065/2063013531
> 2009-01-13 06:56:59,106 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/1731769154213178962, isReference=false, sequence id=8231740, length=255693, majorCompaction=false
> 2009-01-13 06:56:59,129 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/2908303503677637558, isReference=false, sequence id=8485218, length=193497, majorCompaction=false
> 2009-01-13 06:56:59,138 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4011669738439968964, isReference=false, sequence id=2657244, length=628824, majorCompaction=false
> 2009-01-13 06:56:59,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/4385158897188164442, isReference=false, sequence id=7927662, length=277950, majorCompaction=false
> 2009-01-13 06:56:59,168 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5107758633254723438, isReference=false, sequence id=7687719, length=195595, majorCompaction=false
> 2009-01-13 06:56:59,172 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/5309895098547950637, isReference=false, sequence id=8544018, length=34716, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /data/hbase/content/2063013531/url/info/7261360197361933359, isReference=false, sequence id=7468745, length=192133, majorCompaction=false
> 2009-01-13 06:56:59,247 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 7 file(s) in hstore 2063013531/url, max sequence id 8544018
> 2009-01-13 06:56:59,416 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:02,452 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:05,458 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7710643799735487760_4996202 from any node: java.io.IOException: No live nodes contain current block
> 2009-01-13 06:57:08,464 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,466 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening content,0d952ef7d6d31c81552d84ec99c61138,1231777275065
> java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-13 06:57:08,737 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) java.io.IOException: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> ...
> {code}
> Here is master side of things:
> {code}
> 2009-01-13 06:57:08,450 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,810e91fecfad126862a26774bf34df75,1231783565254 from XX.XX.XX.50:60020
> 2009-01-13 06:57:08,734 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: content,0d952ef7d6d31c81552d84ec99c61138,1231777275065: java.io.IOException: Could not obtain block: blk_7710643799735487760_4996202 file=/data/hbase/content/2063013531/url/mapfiles/5309895098547950637/index
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1707)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1535)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
> at org.apache.hadoop.hbase.regionserver.HStore.setupReaders(HStore.java:259)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> from 10.30.94.33:60020
> 2009-01-13 06:57:08,735 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call regionServerReport(address: XX.XX.XX.33:60020, startcode: 1231829713428, load: (requests=0, regions=21, usedHeap=158, maxHeap=1774), [Lorg.apache.hadoop.hbase.HMsg;@17496e57, [Lorg.apache.hadoop.hbase.HRegionInfo;@55eb1db2) from 10.30.94.33:37051: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close. State: name=content,0d952ef7d6d31c81552d84ec99c61138,1231777275065, unassigned=false, pendingOpen=true, open=false, closing=false, pendingClose=false, closed=false, offlined=false
> at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1178)
> at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:823)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:569)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:404)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:374)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:285)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:569)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> This puts region in funky state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.