You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (JIRA)" <ji...@apache.org> on 2014/08/01 00:11:40 UTC
[jira] [Updated] (HBASE-11620) Record the class name of Writer in
WAL header so that only proper Reader can open the WAL file
[ https://issues.apache.org/jira/browse/HBASE-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-11620:
---------------------------
Attachment: 11620-v4.txt
In patch v4, WAL is submitted to HLogSplitter.
For corrupted hlog code path, I haven't found a proper way to verify.
I observed the following in test output:
{code}
2014-07-31 14:51:25,765 WARN [main] wal.HLogSplitter(290): Could not get reader, corrupted log file file:/Users/tyu/trunk/hbase-server/target/test-data/aae07de8-e64e-45fa-b6d6-6ede1bbccc9b/log/hlog.1406843484910
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$CorruptedLogFileException: skipErrors=true Could not open hlog file:/Users/tyu/trunk/hbase-server/target/test-data/aae07de8-e64e-45fa-b6d6-6ede1bbccc9b/log/hlog.1406843484910 ignoring
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:602)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:288)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:231)
at org.apache.hadoop.hbase.regionserver.wal.TestHLogReaderOnSecureHLog.testHLogReaderOnSecureHLog(TestHLogReaderOnSecureHLog.java:123)
...
Caused by: java.io.IOException: Expected ProtobufLogWriter, got SecureProtobufLogWriter
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:140)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:101)
at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:69)
at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:127)
at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:90)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:668)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:577)
... 27 more
2014-07-31 14:51:25,781 WARN [main] wal.HLogSplitter(295): Nothing to split in log file file:/Users/tyu/trunk/hbase-server/target/test-data/aae07de8-e64e-45fa-b6d6-6ede1bbccc9b/log/hlog.1406843484910
2014-07-31 14:51:25,781 DEBUG [main] wal.HLogSplitter(367): Finishing writing output logs and closing down.
2014-07-31 14:51:25,781 INFO [main] wal.HLogSplitter(380): Processed 0 edits across 0 regions; log file=file:/Users/tyu/trunk/hbase-server/target/test-data/aae07de8-e64e-45fa-b6d6-6ede1bbccc9b/log/hlog.1406843484910 is corrupted = true progress failed = false
{code}
However, there was no exception coming out of HLogSplitter.splitLogFile.
> Record the class name of Writer in WAL header so that only proper Reader can open the WAL file
> ----------------------------------------------------------------------------------------------
>
> Key: HBASE-11620
> URL: https://issues.apache.org/jira/browse/HBASE-11620
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.4
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Critical
> Fix For: 0.99.0, 0.98.5, 2.0.0
>
> Attachments: 11620-v1.txt, 11620-v2.txt, 11620-v3.txt, 11620-v4.txt
>
>
> Reported by Kiran in this thread: "HBase file encryption, inconsistencies observed and data loss"
> After step 4 ( i.e disabling of WAL encryption, removing SecureProtobufReader/Writer and restart), read of encrypted WAL fails mainly due to EOF exception at Basedecoder. This is not considered as error and these WAL are being moved to /oldWALs.
> Following is observed in log files:
> {code}
> 2014-07-30 19:44:29,254 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: Splitting hlog: hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017, length=172
> 2014-07-30 19:44:29,254 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: DistributedLogReplay = false
> 2014-07-30 19:44:29,313 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] util.FSHDFSUtils: Recovering lease on dfs file hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017
> 2014-07-30 19:44:29,315 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017 after 1ms
> 2014-07-30 19:44:29,429 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-0] wal.HLogSplitter: Writer thread Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-0,5,main]: starting
> 2014-07-30 19:44:29,429 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-1] wal.HLogSplitter: Writer thread Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-1,5,main]: starting
> 2014-07-30 19:44:29,430 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-2] wal.HLogSplitter: Writer thread Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-2,5,main]: starting
> 2014-07-30 19:44:29,591 ERROR [RS_LOG_REPLAY_OPS-HOST-16:15264-1] codec.BaseDecoder: Partial cell read caused by EOF: java.io.IOException: Premature EOF from inputStream
> 2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: Finishing writing output logs and closing down.
> 2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: Waiting for split writer threads to finish
> 2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: Split writers finished
> 2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1] wal.HLogSplitter: Processed 0 edits across 0 regions; log file=hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017 is corrupted = false progress failed = false
> {code}
> To fix this, we need to propagate EOF exception to HLogSplitter. Any suggestions on the fix?
> -------- (end of quote from Kiran)
> In BaseDecoder#rethrowEofException() :
> {code}
> if (!isEof) throw ioEx;
> LOG.error("Partial cell read caused by EOF: " + ioEx);
> EOFException eofEx = new EOFException("Partial cell read");
> eofEx.initCause(ioEx);
> throw eofEx;
> {code}
> throwing EOFException would not propagate the "Partial cell read" condition to HLogSplitter which doesn't treat EOFException as an error.
> I think IOException should be thrown above - HLogSplitter#getNextLogLine() would translate the IOEx to CorruptedLogFileException.
--
This message was sent by Atlassian JIRA
(v6.2#6252)