You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/09/09 02:15:33 UTC
[jira] Assigned: (HBASE-2967) Failed split: IOE 'File is Corrupt!'
-- sync length not being written out to SequenceFile
[ https://issues.apache.org/jira/browse/HBASE-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-2967:
-----------------------------------------
Assignee: stack
> Failed split: IOE 'File is Corrupt!' -- sync length not being written out to SequenceFile
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-2967
> URL: https://issues.apache.org/jira/browse/HBASE-2967
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.90.0
>
>
> We saw this on one of our clusters:
> {code}
> 2010-09-07 18:07:16,229 WARN org.apache.hadoop.hbase.master.RegionServerOperationQueue: Failed processing: ProcessServerShutdown of sv4borg18,60020,1283516293515; putting onto delayed todo queue
> java.io.IOException: File is corrupt!
> at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1907)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1932)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1837)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1883)
> at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:121)
> at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:113)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.parseHLog(HLog.java:1493)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1256)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1143)
> at org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:299)
> at org.apache.hadoop.hbase.master.RegionServerOperationQueue.process(RegionServerOperationQueue.java:147)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:532)
> {code}
> Because it was an IOE, it got requeued. Each time around we failed on it again.
> A few things:
> + This exception needs to add filename and the position in file at which problem found.
> + Need to commit little patch over in HBASE-2889 that outputs position and ordinal of wal edit because it helps diagnose these kinds of issues.
> + We should be able to skip the bad edit; just postion ourselves at byte past the bad sync and start reading again
> + There must be something about our setup that makes it so we fail write of the sync 16 random bytes that make up the SF 'sync' marker though oddly for one of the files, the sync failure happens at 1/3rd of the way into a 64MB wal, edit #2000 out of 130k odd edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.