[jira] [Resolved] (HBASE-7402) java.io.IOException: Got error in response to OP_READ_BLOCK

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Kyle Purtell resolved HBASE-7402.
----------------------------------------
    Resolution: Invalid

> java.io.IOException: Got error in response to OP_READ_BLOCK
> -----------------------------------------------------------
>
>                 Key: HBASE-7402
>                 URL: https://issues.apache.org/jira/browse/HBASE-7402
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 0.90.4, 0.94.0
>            Reporter: samar
>            Priority: Major
>
> Getting this error on our hbase version 0.90.4-cdh3u3
> 2012-12-18 02:35:39,082 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.x.x.x:xxxxx for file /hbase/table_x/37bea13d03ed9fa611941cc4aad6e8c2/scores/7355825801969613604 for block 3174705353677971357:java.io.IOException: Got error in response to OP_READ_BLOCK self=/x.x.x.x, remote=/x.x.x.x:xxxx for file /hbase/table_x/37bea13d03ed9fa611941cc4aad6e8c2/scores/7355825801969613604 for block 3174705353677971357_1028665
>         at org.apache.hadoop.hdfs.DFSClient$RemoteBlockReader.newBlockReader(DFSClient.java:1673)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.getBlockReader(DFSClient.java:2383)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:2272)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2438)
>         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
>         at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>         at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:141)
>         at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
>         at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>         at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1446)
>         at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1303)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:77)
>         at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1405)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2467)
>         at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1192)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1184)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1168)
>         at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3215)
> this causes the HBase RS to hang and hence stops responding.
>  In NameNode the block was delete before.. ( as per the timestamp)
> 2012-12-18 02:25:19,027 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask x.x.x.x:xxxxx to delete  blk_3174705353677971357_1028665 blk_-9072685530813588257_1028824
> 2012-12-18 02:25:19,027 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask x.x.x.x:xxxxx to delete  blk_5651962510569886604_1028711
> 2012-12-18 02:25:22,027 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask x.x.x.x:xxxxx to delete  blk_3174705353677971357_1028665
> Looks like org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream is cacheing the  block location and causing this issue



--
This message was sent by Atlassian Jira
(v8.20.7#820007)