You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/03/22 03:00:06 UTC
[jira] [Updated] (HBASE-3674) Treat ChecksumException as we would a
ParseException splitting logs; else we replay split on every restart
[ https://issues.apache.org/jira/browse/HBASE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3674:
-------------------------
Attachment: 3674-v2.txt
> Treat ChecksumException as we would a ParseException splitting logs; else we replay split on every restart
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3674
> URL: https://issues.apache.org/jira/browse/HBASE-3674
> Project: HBase
> Issue Type: Bug
> Components: wal
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 0.90.2
>
> Attachments: 3674-v2.txt, 3674.txt
>
>
> In short, a ChecksumException will fail log processing for a server so we skip out w/o archiving logs. On restart, we'll then reprocess the logs -- hit the checksumexception anew, usually -- and so on.
> Here is the splitLog method (edited):
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> ....
> outputSink.startWriterThreads(entryBuffers);
>
> try {
> int i = 0;
> for (FileStatus log : logfiles) {
> Path logPath = log.getPath();
> long logLength = log.getLen();
> splitSize += logLength;
> LOG.debug("Splitting hlog " + (i++ + 1) + " of " + logfiles.length
> + ": " + logPath + ", length=" + logLength);
> try {
> recoverFileLease(fs, logPath, conf);
> parseHLog(log, entryBuffers, fs, conf);
> processedLogs.add(logPath);
> } catch (EOFException eof) {
> // truncated files are expected if a RS crashes (see HBASE-2643)
> LOG.info("EOF from hlog " + logPath + ". Continuing");
> processedLogs.add(logPath);
> } catch (FileNotFoundException fnfe) {
> // A file may be missing if the region server was able to archive it
> // before shutting down. This means the edits were persisted already
> LOG.info("A log was missing " + logPath +
> ", probably because it was moved by the" +
> " now dead region server. Continuing");
> processedLogs.add(logPath);
> } catch (IOException e) {
> // If the IOE resulted from bad file format,
> // then this problem is idempotent and retrying won't help
> if (e.getCause() instanceof ParseException ||
> e.getCause() instanceof ChecksumException) {
> LOG.warn("ParseException from hlog " + logPath + ". continuing");
> processedLogs.add(logPath);
> } else {
> if (skipErrors) {
> LOG.info("Got while parsing hlog " + logPath +
> ". Marking as corrupted", e);
> corruptedLogs.add(logPath);
> } else {
> throw e;
> }
> }
> }
> }
> if (fs.listStatus(srcDir).length > processedLogs.size()
> + corruptedLogs.size()) {
> throw new OrphanHLogAfterSplitException(
> "Discovered orphan hlog after split. Maybe the "
> + "HRegionServer was not dead when we started");
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> splits = outputSink.finishWritingAndClose();
> }
> return splits;
> }
> {code}
> Notice how we'll only archive logs only if we successfully split all logs. We won't archive 31 of 35 files if we happen to get a checksum exception on file 32.
> I think we should treat a ChecksumException the same as a ParseException; a retry will not fix it if HDFS could not get around the ChecksumException (seems like in our case all replicas were corrupt).
> Here is a play-by-play from the logs:
> {code}
> 813572 2011-03-18 20:31:44,687 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 34 of 35: hdfs://sv2borg170:9000/hbase/.logs/sv2borg182,60020,1300384550664/sv2borg182%3A60020.1300461329481, length=150 65662813573 2011-03-18 20:31:44,687 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://sv2borg170:9000/hbase/.logs/sv2borg182,60020,1300384550664/sv2borg182%3A60020.1300461329481
> ....
> 813617 2011-03-18 20:31:46,238 INFO org.apache.hadoop.fs.FSInputChecker: Found checksum error: b[0, 512]=000000cd000000502037383661376439656265643938636463343433386132343631323633303239371d6170695f6163636573735f746f6b656e5f7374 6174735f6275636b65740000000d9fa4d5dc0000012ec9c7cbaf00ffffffff000000010000006d0000005d00000008002337626262663764626431616561366234616130656334383436653732333132643a32390764656661756c746170695f616e64726f69645f6c6f67676564 696e5f73686172655f70656e64696e675f696e69740000012ec956b02804000000000000000100000000ffffffff4e128eca0eb078d0652b0abac467fd09000000cd000000502034663166613763666165333930666332653138346233393931303132623366331d6170695f6163 636573735f746f6b656e5f73746174735f6275636b65740000000d9fa4d5dd0000012ec9c7cbaf00ffffffff000000010000006d0000005d00000008002366303734323966643036323862636530336238333938356239316237386633353a32390764656661756c746170695f61 6e64726f69645f6c6f67676564696e5f73686172655f70656e64696e675f696e69740000012ec9569f1804000000000000000100000000000000d30000004e2066663763393964303633343339666531666461633761616632613964643631331b6170695f6163636573735f746f 6b656e5f73746174735f68
> 813618 org.apache.hadoop.fs.ChecksumException: Checksum error: /blk_7781725413191608261:of:/hbase/.logs/sv2borg182,60020,1300384550664/sv2borg182%3A60020.1300461329481 at 15064576
> 813619 at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
> 813620 at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
> 813621 at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
> 813622 at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
> 813623 at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> 813624 at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1175)
> 813625 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1807)
> 813626 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1859)
> 813627 at java.io.DataInputStream.read(DataInputStream.java:132)
> 813628 at java.io.DataInputStream.readFully(DataInputStream.java:178)
> 813629 at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> 813630 at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> 813631 at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1937)
> 813632 at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1837)
> 813633 at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1883)
> 813634 at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:198)
> 813635 at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:172)
> 813636 at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.parseHLog(HLogSplitter.java:429)
> 813637 at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:262)
> 813638 at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:188)
> 813639 at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:197)
> 813640 at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:181)
> 813641 at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:384)
> 813642 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283)
> 813643 2011-03-18 20:31:46,239 WARN org.apache.hadoop.hdfs.DFSClient: Found Checksum error for blk_7781725413191608261_14589573 from 10.20.20.182:50010 at 15064576
> 813644 2011-03-18 20:31:46,240 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7781725413191608261_14589573 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...
> 813645 2011-03-18 20:31:49,243 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=80624 entries from hdfs://sv2borg170:9000/hbase/.logs/sv2borg182,60020,1300384550664/sv2borg182%3A60020.1300461329481
> ....
> {code}
> See code above. On exception we'll dump edits read so far from this block, close out all writers tying off recovered.edits so far written. We'll skip archiving these files because we only archive if all files are processed; we won't archive files 30 of 35 if we failed splitting on file 31.
> I think checksumexception should be treated same as a ParseException
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira