You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/01/09 21:11:59 UTC
[jira] Created: (HBASE-1115) Regions failed compaction left in open
and writable state but are not scannable
Regions failed compaction left in open and writable state but are not scannable
-------------------------------------------------------------------------------
Key: HBASE-1115
URL: https://issues.apache.org/jira/browse/HBASE-1115
Project: Hadoop HBase
Issue Type: Bug
Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
Reporter: Andrew Purtell
Priority: Critical
After an instance of HBASE-1114, where compaction fails, the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm upgrading to Hadoop 0.19 (finally) this afternoon.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1115) Regions failed compaction left in
open and writable state but are not scannable
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662505#action_12662505 ]
Andrew Purtell commented on HBASE-1115:
---------------------------------------
I've seen the same thing after HBASE-1104. This is a broad symptom of compaction failure I suspect.
> Regions failed compaction left in open and writable state but are not scannable
> -------------------------------------------------------------------------------
>
> Key: HBASE-1115
> URL: https://issues.apache.org/jira/browse/HBASE-1115
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
> Reporter: Andrew Purtell
> Priority: Critical
>
> After an instance of HBASE-1114, where compaction fails, the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
> Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm upgrading to Hadoop 0.19 (finally) this afternoon.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1115) Rollback after regions failed
compaction
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1115:
----------------------------------
Description:
When compaction fails the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
Should there be rollback after a failed compaction somehow?
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
was:
After an instance of HBASE-1114, where compaction fails, the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm upgrading to Hadoop 0.19 (finally) this afternoon.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
Priority: Major (was: Critical)
Summary: Rollback after regions failed compaction (was: Regions failed compaction left in open and writable state but are not scannable)
Edited issue title to broaden, marked down
> Rollback after regions failed compaction
> ----------------------------------------
>
> Key: HBASE-1115
> URL: https://issues.apache.org/jira/browse/HBASE-1115
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
> Reporter: Andrew Purtell
>
> When compaction fails the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
> Should there be rollback after a failed compaction somehow?
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1115) Regions failed compaction left in
open and writable state but are not scannable
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662574#action_12662574 ]
stack commented on HBASE-1115:
------------------------------
I can't say for certain but seems related to hbase-1114 Andrew? The FNFE is root of the above. Seemed to start after the NPEs.
> Regions failed compaction left in open and writable state but are not scannable
> -------------------------------------------------------------------------------
>
> Key: HBASE-1115
> URL: https://issues.apache.org/jira/browse/HBASE-1115
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
> Reporter: Andrew Purtell
> Priority: Critical
>
> After an instance of HBASE-1114, where compaction fails, the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
> Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm upgrading to Hadoop 0.19 (finally) this afternoon.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1115) Regions failed compaction left in
open and writable state but are not scannable
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662610#action_12662610 ]
Andrew Purtell commented on HBASE-1115:
---------------------------------------
Yes is definitely related to HBASE-1114. But should there be rollback after failed compaction? This also happened in conjunction with HBASE-1104.
> Regions failed compaction left in open and writable state but are not scannable
> -------------------------------------------------------------------------------
>
> Key: HBASE-1115
> URL: https://issues.apache.org/jira/browse/HBASE-1115
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell cluster, HBase TRUNK on hadoop 0.18 branch
> Reporter: Andrew Purtell
> Priority: Critical
>
> After an instance of HBASE-1114, where compaction fails, the affected region is left in an open and writable state, but scanners fail construction. Later a manual reassignment via close_region brings the region all the way back up.
> Line numbers are for 0.19-on-hadoop-0.18 branch, sorry about that. I'm upgrading to Hadoop 0.19 (finally) this afternoon.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.50:60020 for region content,c84bbfc94b2143e41ba119d159be2958,1231518442461, row 'c84bbfc94b2143e41ba119d159be2958', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2119)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1878)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1162)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1673)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1707725801/url/mapfiles/7039742044868774100/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1431)
> at org.apache.hadoop.hbase.io.SequenceFile$Reader.(SequenceFile.java:1426)
> at org.apache.hadoop.hbase.io.MapFile$Reader.createDataFileReader(MapFile.java:310)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:96)
> at org.apache.hadoop.hbase.io.MapFile$Reader.open(MapFile.java:292)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:79)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:65)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.