You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/12/04 20:14:44 UTC
[jira] Created: (HBASE-1046) NPE during compaction/split breaks
table
NPE during compaction/split breaks table
----------------------------------------
Key: HBASE-1046
URL: https://issues.apache.org/jira/browse/HBASE-1046
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Reporter: Andrew Purtell
Fix For: 0.19.0
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
The previous region server was asked by the master to close it:
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
2008-12-04 07:58:03,526 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4580f7292ceb2f6c8d732d935302633,1228328680554
There are no exceptions or other indications of problems with the region or the region server in the relevant log file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman resolved HBASE-1046.
----------------------------------
Resolution: Fixed
Fixed by HBASE-543
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653492#action_12653492 ]
stack commented on HBASE-1046:
------------------------------
See this for server 32:
{code}
2008-12-04 11:17:21,541 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.30.94.32:60020
2008-12-04 11:17:22,258 INFO org.apache.hadoop.hbase.master.ServerManager: Cancelling lease for 10.30.94.32:60020
2008-12-04 11:18:52,722 INFO org.apache.hadoop.hbase.master.ServerManager: Received start message from: 10.30.94.32:60020
{code}
So that explains why region was assigned to 49.
There is definetly a hiccup in here. How does master get to know of new region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 before it gets the split message? Once assigned, why is it assigned anew after split processing?
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1046) NPE during compaction/split breaks
table
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1046:
----------------------------------
Description:
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
This is the region history:
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
On 10.30.94.49:
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
On 10.30.94.54:
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
was:
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
This is the region history:
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
On 10.30.94.49:
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
On 10.30.94.54:
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
This all looks fine to me. There are no exceptions or errors during the split.
> NPE during compaction/split breaks table
> ----------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1046) NPE during compaction/split breaks
table
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1046:
----------------------------------
Description:
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
This is the region history:
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
On 10.30.94.49:
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
On 10.30.94.54:
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
This all looks fine to me. There are no exceptions or errors during the split.
was:
After reassignment of region split, scanners (mapred tasks) fail:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
... 10 more
According to the master the regions around this row are:
content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
On 10.30.94.49, just before the HStoreScanner exceptions begin:
2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
The previous region server was asked by the master to close it:
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Worker: MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
2008-12-04 07:58:03,526 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4580f7292ceb2f6c8d732d935302633,1228328680554
There are no exceptions or other indications of problems with the region or the region server in the relevant log file.
> NPE during compaction/split breaks table
> ----------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
> This all looks fine to me. There are no exceptions or errors during the split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1046) NPE during
compaction/split breaks table
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653428#action_12653428 ]
apurtell edited comment on HBASE-1046 at 12/4/08 11:41 AM:
-----------------------------------------------------------------
I think 10.30.94.49 and 10.30.94.54 tried to compact the same region at roughly the same time.
On 10.30.94.49:
2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
java.lang.NullPointerException
On 10.30.94.54:
2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
was (Author: apurtell):
I think 10.30.94.49 and 10.30.94.54 tried to compact the same region at the same time.
> NPE during compaction/split breaks table
> ----------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657259#action_12657259 ]
stack commented on HBASE-1046:
------------------------------
Thing to try: Add a pause to the HRS -- into CompactSplitThread around line #186 -- after it updates .META. with split info. Make pause > master metascan period. Are regions assigned twice?
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1046) NPE during compaction/split breaks
table
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1046:
----------------------------------
Priority: Blocker (was: Major)
Marking as blocker. Seems quite serious.
> NPE during compaction/split breaks table
> ----------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> The previous region server was asked by the master to close it:
> 2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
> 2008-12-04 07:58:03,525 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_CLOSE: content,c4580f7292ceb2f6c8d732d935302633,1228328680554: Overloaded
> 2008-12-04 07:58:03,526 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4580f7292ceb2f6c8d732d935302633,1228328680554
> There are no exceptions or other indications of problems with the region or the region server in the relevant log file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658194#action_12658194 ]
Jim Kellerman commented on HBASE-1046:
--------------------------------------
Yes, I have been able to reproduce it.
Hopefully I'll have a patch later today.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) NPE during compaction/split breaks
table
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653428#action_12653428 ]
Andrew Purtell commented on HBASE-1046:
---------------------------------------
I think 10.30.94.49 and 10.30.94.54 tried to compact the same region at the same time.
> NPE during compaction/split breaks table
> ----------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-1046:
---------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HBASE-543
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658188#action_12658188 ]
stack commented on HBASE-1046:
------------------------------
Any progress on this or 1051?
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653485#action_12653485 ]
stack commented on HBASE-1046:
------------------------------
Looking at code, I cannot see how opens would be processed after splits though evidence presented above would indicate this is happening. Here is full lifecycle of region listed above from log Andrews sent:
{code}
2008-12-04 02:54:23,504 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 to server 10.30.94.32:60020
2008-12-04 02:54:26,831 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.32:60020
2008-12-04 02:54:29,875 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.32:60020
2008-12-04 02:54:32,882 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.32:60020
2008-12-04 02:54:35,905 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.32:60020
2008-12-04 02:54:38,924 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.32:60020
2008-12-04 02:54:38,925 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 open on 10.30.94.32:60020
2008-12-04 02:54:38,925 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in region .META.,,1 with startcode 1228377259645 and server 10.30.94.32:60020
2008-12-04 11:17:39,549 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 to server 10.30.94.49:60020
2008-12-04 11:17:42,578 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.49:60020
2008-12-04 11:17:45,595 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.49:60020
2008-12-04 11:17:45,595 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 open on 10.30.94.49:60020
2008-12-04 11:17:45,595 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
2008-12-04 11:18:25,482 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 to server 10.30.94.54:60020
2008-12-04 11:18:28,501 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.54:60020
2008-12-04 11:18:28,501 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.54:60020
2008-12-04 11:18:28,503 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 open on 10.30.94.54:60020
2008-12-04 11:18:28,503 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in region .META.,,1 with startcode 1228377257408 and server 10.30.94.54:60020
2008-12-04 11:18:30,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 split; daughters: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257 from 10.30.94.49:60020
2008-12-04 11:18:30,539 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 to server 10.30.94.49:60020
2008-12-04 11:18:33,548 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.49:60020
2008-12-04 11:18:33,549 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.49:60020
2008-12-04 11:18:33,550 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 open on 10.30.94.49:60020
2008-12-04 11:18:33,550 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
2008-12-04 11:19:25,439 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 because daughter splits no longer hold references
{code}
Trying to figure out what happed to host 32 around 11:17:39,549. Log is at INFO but should see server lease expired or MSG_REGION_CLOSE if they came in.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-1046:
------------------------------------
Assignee: Jim Kellerman
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653449#action_12653449 ]
Andrew Purtell commented on HBASE-1046:
---------------------------------------
>From the master log you would think that RegionManager assigned out a daughter region 5 seconds before ServerManager processed the MSG_REPORT_SPLIT that informed the master of the daughter in the first place.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654022#action_12654022 ]
Andrew Purtell commented on HBASE-1046:
---------------------------------------
I think it happened again after regionserver went down with oome and the region was reassigned twice. I sent the master and region server logs at DEBUG level to stack at his gmail address. I'm still on the road so did not have time to look them over in any detail.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653446#action_12653446 ]
Andrew Purtell commented on HBASE-1046:
---------------------------------------
Both daughter regions are damaged. Both seem to be multiply assigned.
Everything relevant from the master log:
For content,c4e518dc579ca98bf97ea13d1f7c80fc
2008-12-04 06:17:39,549 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 to server 10.30.94.49:60020
2008-12-04 06:17:42,578 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.49:60020
2008-12-04 06:17:45,595 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 from 10.30.94.49:60020
2008-12-04 06:17:45,595 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 open on 10.30.94.49:60020
2008-12-04 06:17:45,595 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
2008-12-04 06:18:25,482 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 to server 10.30.94.54:60020
2008-12-04 06:18:28,501 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.54:60020
2008-12-04 06:18:28,502 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c528c8864e67a3addcb50587586dfe38,1228407504257 in region .META.,,1 with startcode 1228377257408 and server 10.30.94.54:60020
2008-12-04 06:18:28,503 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 open on 10.30.94.54:60020
2008-12-04 06:18:28,503 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in region .META.,,1 with startcode 1228377257408 and server 10.30.94.54:60020
2008-12-04 06:18:30,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 split; daughters: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257 from 10.30.94.49:60020
2008-12-04 06:18:30,539 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 to server 10.30.94.49:60020
2008-12-04 06:18:33,548 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 from 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c528c8864e67a3addcb50587586dfe38,1228407504257 open on 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c528c8864e67a3addcb50587586dfe38,1228407504257 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
2008-12-04 06:18:33,550 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 open on 10.30.94.49:60020
2008-12-04 06:18:33,550 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
2008-12-04 06:19:25,439 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 because daughter splits no longer hold references
For content,c528c8864e67a3addcb50587586dfe38:
2008-12-04 06:18:25,480 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c528c8864e67a3addcb50587586dfe38,1228407504257 to server 10.30.94.54:60020
2008-12-04 06:18:28,501 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257 from 10.30.94.54:60020
2008-12-04 06:18:28,501 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c528c8864e67a3addcb50587586dfe38,1228407504257 open on 10.30.94.54:60020
2008-12-04 06:18:28,502 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c528c8864e67a3addcb50587586dfe38,1228407504257 in region .META.,,1 with startcode 1228377257408 and server 10.30.94.54:60020
2008-12-04 06:18:30,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 split; daughters: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257 from 10.30.94.49:60020
2008-12-04 06:18:30,538 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region content,c528c8864e67a3addcb50587586dfe38,1228407504257 to server 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257 from 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: content,c528c8864e67a3addcb50587586dfe38,1228407504257 open on 10.30.94.49:60020
2008-12-04 06:18:33,549 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row content,c528c8864e67a3addcb50587586dfe38,1228407504257 in region .META.,,1 with startcode 1228396338736 and server 10.30.94.49:60020
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658192#action_12658192 ]
stack commented on HBASE-1046:
------------------------------
Have you been able to reproduce? Any ETA on when done? Thanks.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655050#action_12655050 ]
Andrew Purtell commented on HBASE-1046:
---------------------------------------
I think an important part of the recipe to reproduce is artificial loading of DFS and of the CPU of the master node.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658189#action_12658189 ]
Jim Kellerman commented on HBASE-1046:
--------------------------------------
Working on them.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1046:
----------------------------------
Summary: Region assigned to two regionservers after split (was: NPE during compaction/split breaks table)
Edited the title of this issue to reflect analysis findings.
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1046) Region assigned to two regionservers
after split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658196#action_12658196 ]
stack commented on HBASE-1046:
------------------------------
Good stuff
> Region assigned to two regionservers after split
> ------------------------------------------------
>
> Key: HBASE-1046
> URL: https://issues.apache.org/jira/browse/HBASE-1046
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Andrew Purtell
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.19.0
>
>
> After reassignment of region split, scanners (mapred tasks) fail:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
> at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
> at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
> at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
> at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
> at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
> ... 10 more
> According to the master the regions around this row are:
> content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38
> On 10.30.94.49, just before the HStoreScanner exceptions begin:
> 2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> 2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
> This is the region history:
> content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
> Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
> Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
> Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> On 10.30.94.49:
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
> 2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
> tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
> 2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
> 2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
> 2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
> REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
> 91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec
> On 10.30.94.54:
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
> Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
> 2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
> 2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
> 2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.