[jira] [Commented] (HBASE-11625) Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum

["Brahma Reddy Battula (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998095#comment-14998095 ] 

Brahma Reddy Battula commented on HBASE-11625:
----------------------------------------------

[~apurtell]/[~syuanjiang]/[~ndimiduk]/[~qian wang]/[~ram_krish]

Any update on this ..? Same issue we faced in production environment and we are using hbase1.0 and hadoop2.6.

Hfile replcation is 2,one good replica and one corrupted replica. Noticed following.

 *HDFS client:*  When using hdfs connection client reads HFile A to Bad DataNode (where data is corrupted), the DataNode reported checksum error,
hdfs client attempts to read data from another DataNode, and read success.
---DFSClient with hdfs check sum

 *RegionServer:*  RegionServer closing the hdfs clients, no check sum hdfs enabled client reads HFile A.
---then use DFSClient without hdfs check sum.

Please correct me If I am wrong..

Bytheway following is the trace. *Major compaction failed.* 
{noformat}
2015-09-16 13:03:07,307 | INFO  | regionserver21302-smallCompactions-1441158778210 | Starting compaction of 6 file(s) in d of TB_HTTP_201509,820,1441041221494.1c80ce3eddc7b463b1f9525d2f440798. into tmpdir=hdfs://hacluster/hbase/data/default/TB_HTTP_201509/1c80ce3eddc7b463b1f9525d2f440798/.tmp, totalSize=22.6 M | org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1077)
2015-09-16 13:03:07,311 | ERROR | regionserver21302-smallCompactions-1441158778210 | Compaction failed Request = regionName=TB_HTTP_201509,820,1441041221494.1c80ce3eddc7b463b1f9525d2f440798., storeName=d, fileCount=6, fileSize=22.6 M (3.7 M, 3.8 M, 3.8 M, 3.8 M, 3.8 M, 3.8 M), priority=-1701, time=10101672953052926 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:510)
java.io.IOException: Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs://hacluster/hbase/data/default/TB_HTTP_201509/1c80ce3eddc7b463b1f9525d2f440798/d/63cc160d266748599d338b7c9e390a23, compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false][prefetchOnOpen=false], firstKey=8200005291_460000132556183_9223372035413272744_1919/d:aa/1441530258037/Put, lastKey=8249864621_460020161249953_9223372035413273715_6377/d:di/1441530155261/Put, avgKeyLen=65, avgValueLen=5, entries=319088, length=3964840, cur=null] to key /d:/LATEST_TIMESTAMP/DeleteFamily/vlen=0/mvcc=0
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:164)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:240)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:202)
	at org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:257)
	at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
	at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:109)
	at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1086)
	at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1480)
	at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:495)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00
	at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
	at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:165)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:239)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1486)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:392)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1090)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
	... 12 more
{noformat}

> Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum 
> -----------------------------------------------------------------------------------------
>
>                 Key: HBASE-11625
>                 URL: https://issues.apache.org/jira/browse/HBASE-11625
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 0.94.21, 0.98.4, 0.98.5
>            Reporter: qian wang
>         Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz
>
>
> when using hbase checksum,call readBlockDataInternal() in hfileblock.java, it could happen file corruption but it only can switch to hdfs checksum inputstream till validateBlockChecksum(). If the datablock's header corrupted when b = new HFileBlock(),it throws the exception "Invalid HFile block magic" and the rpc call fail



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)