You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2010/03/01 22:05:10 UTC
Cannot open filename error
Hi,
I saw this in our HBase 0.20.1 master log:
2010-03-01 12:38:42,451 INFO [HMaster] master.ProcessRegionOpen(80):
Updated row domaincrawltable,,1267475905927 in region .META.,,1 with
startcode=1267475746189, server=10.10.31.135:60020
2010-03-01 12:39:06,088 INFO [Thread-10]
master.ServerManager$ServerMonitor(130): 3 region servers, 0 dead, average
load 17.666666666666668
2010-03-01 12:39:25,939 INFO [RegionManager.rootScanner]
master.BaseScanner(146): RegionManager.rootScanner scanning meta region
{server: 10.10.31.136:60020, regionname: -ROOT-,,0, startKey: <>}
2010-03-01 12:39:25,944 INFO [RegionManager.rootScanner]
master.BaseScanner(224): RegionManager.rootScanner scan of 1 row(s) of meta
region {server: 10.10.31.136:60020, regionname: -ROOT-,,0, startKey: <>}
complete
2010-03-01 12:39:32,053 WARN [RegionManager.metaScanner]
master.MetaScanner(78): Scan one META region: {server: 10.10.31.137:60020,
regionname: .META.,,1, startKey: <>}
java.io.IOException: java.io.IOException: Cannot open filename
/hbase/.META./1028785192/info/4094183944991075010
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1474)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1800)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1616)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1743)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:935)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1147)
at
org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
at
org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1471)
at
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2294)
at
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2283)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
[root@snv-it-lin-006 software]# hfs -ls /hbase/.META./1028785192/info/
Found 4 items
-rw-r--r-- 3 hbaseadmin supergroup 4465 2010-03-01 12:35
/hbase/.META./1028785192/info/2734607187585444172
-rw-r--r-- 3 hbaseadmin supergroup 14140 2010-03-01 10:41
/hbase/.META./1028785192/info/6809882943741425603
-rw-r--r-- 3 hbaseadmin supergroup 226422 2010-03-01 10:41
/hbase/.META./1028785192/info/7072901656043692885
-rw-r--r-- 3 hbaseadmin supergroup 14403 2010-03-01 12:36
/hbase/.META./1028785192/info/7408853062491968424
What could have led to missing file under .META. ?
Thanks
Re: Cannot open filename error
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Ted,
Most probably double assignment, have a look at the region server logs
and you will see that a some point the META was compacted while the
another region server was hosting it. This usually happens under
stress and is mostly fixed in 0.20.3 then there's
http://issues.apache.org/jira/browse/HBASE-2244 that we hope to fix
for 0.20.4
J-D
On Mon, Mar 1, 2010 at 1:05 PM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> I saw this in our HBase 0.20.1 master log:
>
> 2010-03-01 12:38:42,451 INFO [HMaster] master.ProcessRegionOpen(80):
> Updated row domaincrawltable,,1267475905927 in region .META.,,1 with
> startcode=1267475746189, server=10.10.31.135:60020
> 2010-03-01 12:39:06,088 INFO [Thread-10]
> master.ServerManager$ServerMonitor(130): 3 region servers, 0 dead, average
> load 17.666666666666668
> 2010-03-01 12:39:25,939 INFO [RegionManager.rootScanner]
> master.BaseScanner(146): RegionManager.rootScanner scanning meta region
> {server: 10.10.31.136:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-03-01 12:39:25,944 INFO [RegionManager.rootScanner]
> master.BaseScanner(224): RegionManager.rootScanner scan of 1 row(s) of meta
> region {server: 10.10.31.136:60020, regionname: -ROOT-,,0, startKey: <>}
> complete
> 2010-03-01 12:39:32,053 WARN [RegionManager.metaScanner]
> master.MetaScanner(78): Scan one META region: {server: 10.10.31.137:60020,
> regionname: .META.,,1, startKey: <>}
> java.io.IOException: java.io.IOException: Cannot open filename
> /hbase/.META./1028785192/info/4094183944991075010
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1474)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1800)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1616)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1743)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:981)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:935)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1147)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
> at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1471)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2294)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2283)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>
> [root@snv-it-lin-006 software]# hfs -ls /hbase/.META./1028785192/info/
> Found 4 items
> -rw-r--r-- 3 hbaseadmin supergroup 4465 2010-03-01 12:35
> /hbase/.META./1028785192/info/2734607187585444172
> -rw-r--r-- 3 hbaseadmin supergroup 14140 2010-03-01 10:41
> /hbase/.META./1028785192/info/6809882943741425603
> -rw-r--r-- 3 hbaseadmin supergroup 226422 2010-03-01 10:41
> /hbase/.META./1028785192/info/7072901656043692885
> -rw-r--r-- 3 hbaseadmin supergroup 14403 2010-03-01 12:36
> /hbase/.META./1028785192/info/7408853062491968424
>
> What could have led to missing file under .META. ?
>
> Thanks
>