[jira] Updated: (HBASE-2643) Figure how to deal with eof splitting logs

["Todd Lipcon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HBASE-2643:
-------------------------------

    Description: 
When splitting the WAL and encountering EOF, it's not clear what to do. Initial discussion of this started in http://review.hbase.org/r/74/ - summarizing here for brevity:

We can get an EOFException while splitting the WAL in the following cases:
- The writer died after creating the file but before even writing the header (or crashed halfway through writing the header)
- The writer died in the middle of flushing some data - sync() guarantees that we can see _at least_ the last edit, but we may see half of an edit that was being written out when the RS crashed (especially for large rows)
- The data was actually corrupted somehow (eg a length field got changed to be too long and thus points past EOF)

Ideally we would know when we see EOF whether it was really the last record, and in that case, simply drop that record (it wasn't synced, so therefore we dont need to split it). Some open questions:
  - Currently we ignore empty files. Is it ok to ignore an empty log file if it's not the last one?
  - Similarly, do we ignore an EOF mid-record if it's not the last log file?

  was:
During review of 2437, lots of discussion around how to deal with eof.  This issue is about making decision about how to do eof treatment when reading wals.

Below is copied from http://review.hbase.org/r/74/

{code}
yes, I think so. The RS could have crashed right after opening but before writing any data, and if the master failed to recover that, then we'd never recover that region. I say ignore with a WARN
Cosmin Lehene 6 days, 18 hours ago (May 25th, 2010, 2:27 p.m.)
more aspects here:
I think the reported size will be >0 after recover, even if file has no records. I was asking if we should add logic to check if it's the last log. 
EOF for non zero length, non zero records file means file is corrupted. 
Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:27 p.m.)
I agree if it has no records (I think - do we syncfs after writing the sequencefile header?). But there's the case where inside SequenceFile we call create, but never actually write any bytes. This is still worth recovering.

In general I think a corrupt tail means we should drop that record (incomplete written record) but not shut down. This is only true if it's the tail record, though.
Cosmin Lehene 3 days ago (May 29th, 2010, 8:56 a.m.)
 - How can we determine it's the tail record or the 5th out of 10 records that's broken? We just get an EOF when calling next()
 - Currently we ignore empty files. Is it ok to ignore an empty log file if it's not the last one?
 - I'm not sure whether it's possible to get an EOF when acquiring the reader for a file after it has been recoverFileLease()-ed. So the whole try/catch for HLog.getReader might be redundant.

When reading log entries we currently don't catch. We read as much as we can and then let any exception bubble up. splitLog logic decides what to do next: If we get to a broken record it will most probably throw an EOF in there and based on skip.errors setting it will act accordingly. There will be no EOF if there are no records, though and we continue.

There are two possible reasons for a file being corrupted/empty:
1 HRegion died => only the last log entry (edit) in the last log in the directory should be affected => we could continue but are we sure it's the tail record?
2 Another component screwed things up (bug) => other logs than the last one could be affected => we should halt in this situation.
Add comment
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java (Diff revision 1)
1455	
        throw e;
see above logic - writer could have crashed after writing only part of the sequencefile header, etc, so we should just warn and continue
Cosmin Lehene 6 days, 18 hours ago (May 25th, 2010, 2:27 p.m.)
see above comment
Add comment
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java (Diff revision 1)
1471	
    } finally {
I think we need to handle EOF specially here too, though OK to leave this for another JIRA. IIRC one of the FB guys opened this already
Cosmin Lehene 6 days, 18 hours ago (May 25th, 2010, 2:30 p.m.)
what's the other JIRA? see my above comments.
Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:27 p.m.)
Can't find it now... does my above comment make sense?
{code}


Updated description to a summary of the issue at hand - refer back to original review for original discussion.

> Figure how to deal with eof splitting logs
> ------------------------------------------
>
>                 Key: HBASE-2643
>                 URL: https://issues.apache.org/jira/browse/HBASE-2643
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> When splitting the WAL and encountering EOF, it's not clear what to do. Initial discussion of this started in http://review.hbase.org/r/74/ - summarizing here for brevity:
> We can get an EOFException while splitting the WAL in the following cases:
> - The writer died after creating the file but before even writing the header (or crashed halfway through writing the header)
> - The writer died in the middle of flushing some data - sync() guarantees that we can see _at least_ the last edit, but we may see half of an edit that was being written out when the RS crashed (especially for large rows)
> - The data was actually corrupted somehow (eg a length field got changed to be too long and thus points past EOF)
> Ideally we would know when we see EOF whether it was really the last record, and in that case, simply drop that record (it wasn't synced, so therefore we dont need to split it). Some open questions:
>   - Currently we ignore empty files. Is it ok to ignore an empty log file if it's not the last one?
>   - Similarly, do we ignore an EOF mid-record if it's not the last log file?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.